Item Arrangement Research Articles

Implications for Altering the Context in Which Test Items Appear: A Historical Perspective on an Immediate Concern

Research into the effects of item arrangement has been motivated by the need to know the potential effects on item statistics of different item arrangement schemes. This review of the literature confirms that many of the salient and common features of the research can be identified as a function of the practical psychometric concerns of the time. The studies are separated into three periods. The earliest studies investigated the simple main effect of item order on test performance; the late 1960s reflected a change in emphasis to a design that included interactions between item order and factors of examinees’ psychological and biological characteristics; current concern with test disclosure and development of individual adaptive testing instruments has shifted the emphasis to the effects of item order on the stability of item parameters. The literature has produced evidence of context effects, but has not demonstrated that the effects are so strong as to invalidate test theory or practice that is dependent on an assumption of item parameter invariance.

Effects of Item Arrangement, Sex of the Subject, and Test Anxiety on Cognitive and Self-Perception Scores in a Nonquantitative Content Area

Previous research using an examination with a quantitative emphasis revealed a significant sex effect and an item arrangement x sex interaction for a cognitive variable (number right) and an item arrangement effect for self-perception scores pertaining to difficulty, fairness, and anticipated performance on the test. Scores representing number of items right and self-perceptions were analyzed for nonquantitative examination that was assembled into three forms: easy-hard, spiral cyclical, random. Multivariate ANCOVA revealed no significant effects (α = .05) for the cognitive measure. However, significant (α = .05) sex and sex x order effects were found for perceptions scores which were not parallel to those reported previously for a quantitative examination. Implications for the valid interpretation of test scores also are briefly discussed.

Item Arrangement, Cognitive Entry Characteristics, Sex, and Test Anxiety as Predictors of Achievement Examination Performance

This study’s general research question was: Given male and female students in an introductory educational psychology course who vary in cognitive entry characteristics and test anxiety, how do three item arrangements (easy to difficult, difficult to easy, and random) located within a 50-item multiple-choice achievement examination influence students’ total test performance? Two hierarchical multiple regression analyses were used to analyze the data. The four predictor variables and their interactions were tested for the amount of variation that they explained in the dependent variable. The main finding within the context of this study is that item arrangements based on item difficulties do not influence achievement examination performance.

Differential Performance of Males and Females on Easy to Hard Item Arrangements: Influence of Feedback at the Item Level

Differential test performance by males and females was reported on a quantitative examination as a function of item arrangement (Plake, Ansorge, Parker, and Lowry, 1982). Males performed at a higher level than females on tests the items of which were arranged from easy to hard. Plake and Ansorge (1982) speculated that this male superiority may be a function of differential reinforcement in backgrounds of males and females. Using a Latin Square design, this study provided item performance feedback for a nonquantitative examination. Significant sex-by-order effects did not occur, but the lack of this effect could possibly have been due to the use of a nonquantitative examination (Plake and Ansorge, 1983). It still remains to be seen whether, in using a quantitative examination, differential effects of item feedback may be a source of explanation for evidence of sex-by-order effects. Implications for validity of test score interpretation are considered briefly.

Subjective correlation and the regular-random numerosity illusion.

It has been shown previously that a regular pattern of dots appears more numerous than a random pattern with the same physical number. Birnbaum and Veit's theory of contrast with expectancy has been suggested as a possible interpretation of this illusion. In order to test this theory, three groups (N = 102) were given prior exposure to stimuli with either a positive, negative, or zero correlation between regularity and number. The predicted interaction between correlation and item arrangement was not found. The main effect of regularity was equally strong in all three groups.

EFFECTS OF ITEM ARRANGEMENT, KNOWLEDGE OF ARRANGEMENT TEST ANXIETY AND SEX ON TEST PERFORMANCE

The evaluation of student achievement and aptitude by means of paper and pencil multiple-choice tests is confounded by the presence of test idiosyncracies and student characteristics which may impede the accurate assessment of student performance (Millman, Bishop, & Ebel, 1965). Among test characteristics which may affect test performance are the presentation of the questions and the specificity of the test directions. Item arrangement, or the physical location of the test questions within the test is one aspect of item presentation that has been subjected to empirical investigation. Basically, the following four item arrangements have been studied: easy-hard, hard-easy, uniform or spiral-cyclical, and random. Results have generally supported easy-hard ordering for speeded tests, but have shown no practical difference for power tests (Flaugher, Melton, & Meyer, 1968; Brenner, 1964). Specificity of test directions, in particular information provided to the examinee about the item ordering, has been hypothesized to affect test performance (Cronbach, 1946; Hambleton & Traub, 1974). Little empirical evidence, however, has been found to support this idea (Plake, 1980; Plake, Thompson, & Lowry, 1981). Student characteristics which may affect test performance include test wiseness and test anxiety. Studies combining test anxiety and item arrangement show conflicting results (Hambleton & Traub, 1974; Marso, 1970; Smouse & Munz, 1968; Plake, Thompson, & Lowry, 1981). Sarnacki (1979), in his review of factors affecting test performance in the cognitive domain, suggests that familiarity with the arrangement of the items in a test may reduce one's test anxiety. Plake, Thompson, and Lowry (1981) investigated the effects of item arrangement (easy-hard, uniform, random), knowledge of arrangement, and test anxiety on performance on a 48-item mathematics test. No significant (p > .05) order or knowledge effects were found. However, the subject pool consisted of college sophomore volunteers from a general educational psychology class who may not have been motivated to achieve on the mathematics test. Therefore, the validity and applicability of the results may be questionable. The sex of the subject has also been suggested as a variable which will influence test performance, particularly in the area of mathematics (Tobias & Weissbrod, 1980). The purposes of the present study were (1) to investigate the effects of item arrangement, knowledge of arrangement, and test anxiety using motivated subjects to contrast with the Plake, Thompson, and Lowry (1981) research results on non-motivated subjects, and (2) to investigate the interactive effects of sex of the subject and test anxiety, item arrangement and/or knowledge of arrangement on test performance.

Effect of Item Arrangement, Knowledge of Arrangement, and Test Anxiety on Two Scoring Methods

Number right (NR) and elimination (EL) scores were analyzed on 48-item college level mathematics exam that was assembled from pretest data in three forms by varying the item orderings: easy-hard, uniform, random. One-half of the forms contained information explaining the item arrangement and suggesting strategies for taking the test. Several anxiety measures were administered in a balanced order along with the experimental forms to 97 undergraduates. Multivariate ANOVA revealed no significant effects of the independent variables (order, knowledge, anxiety condition) on either cognitive dependent measure (NR or EL) (a = .05). Multivariate ANCOVA, using measures of anxiety as multiple covariates, also failed to show any significant influence (a = .05) on the cognitive dependent measures or a set of perceptional dependent measures. Results suggest that neither test scores nor attitudes are influenced by item order, knowledge thereof, or anxiety level, although factors contributing to the non-significance may have been inherent in the research design because of inadequate power and/or student performance motivation.

Item Arrangement and Knowledge of Arrangement on Test Scores

Three-item orderings (easy-to-hard, spiral cyclical, and random) and two levels of knowledge of ordering (informed and non-informed) were used to study differences in test results, student’s perception of the test’s fairness and difficulty, and student’s estimation of test performance. No significant order effect was found (a = .05); results are consistent with research which have found no difference in test scores when different orderings are used. No significant effects were found for knowledge of the orderings or for the interaction of knowledge of arrangement and order (a = .05). Knowledge, or lack thereof, of item arrangement is therefore not a factor influencing test scores or student’s perceptions about the test for any of the orderings studied.

A Developmental Study of the Regular-Random Numerosity Illusion

Perceived number is influenced by item arrangement. It has been shown previously with adults that a regular pattern of dots appears more numerous than a random pattern with the same physical number. The present study used similar stimuli: two sets of 37 dots arranged within circular areas, one regular and the other random. Forty-eight grade school students were tested; they ranged in age from 5.6 to 14.8 years and included 27 girls and 21 boys. The illusion was found to increase in strength from the second to the eighth grade. The results were discussed in relation to Birnbaum and Veit's theory of contrast with expectancy.

Technical characteristics of the peabody individual achievement test as a function of item arrangement and basal and ceiling rules

The effects that item order and basal and ceiling rules have on test means, variances, and internal consistency estimates for the PIAT mathematics and reading recognition subtests were examined. Seven items on the math subtest and one item on the reading recognition subtest were significantly easier or harder than their test placement indicated. The use of basal and ceiling rules had a pronounced effect on the means, variances, and reliabilities on the multiple choice math subtest, while the rules' effects on the reading recognition subtests were minor. Item order also affected scores on the math subtest.

Perceived Numerosity, Item Arrangement, and Expectancy

Ten sets of dots ranging in number from 7 to 91 per set were presented to 54 subjects. Half were arranged regularly and half randomly. The regular sets were overestimated by an average of 22%, while the random were underestimated by 16% compared with the physical number. Evidence was presented showing that subjects expect results in the opposite direction. The findings are discussed in relation to the expectancy-contrast model of Birnbaum and Veit.

Effect of Item Arrangement on Perceived Numerosity: Randomness vs Regularity

Subjects were shown pairs of dot patterns, a regular one and a randomly arranged set, and were asked to say which appeared more numerous. In three experiments using 53 college students regular patterns appeared significantly more numerous than random arrays, with an average illusion of 5.5%. The results were discussed in relation to the breakdown of conservation of number.

The Effect of Item Arrangement on Classroom Test Performance: Implications for Content Validity

The Effect of Item Arrangement on Classroom Test Performance: Implications for Content Validity

Item arrangement effects on transfer and serial position errors in part-whole learning of different materials.

Item arrangement effects on transfer and serial position errors in part-whole learning of different materials.

THE SELF‐SCORING FLEXILEVEL TEST1

Modifications of administration and item arrangement of a conventional test can force a match between item difficulty levels and the ability level of the examinee. Although different examinees take different sets of items, the scoring method provides comparable scores for all. Furthermore, the test is self‐scoring. These advantages are obtained without some of the usual disadvantages of tailored testing.

Test Statistics as a Function of Item Arrangement

Using a technique that controlled exposure of items, the investigator examined the effect on mean test score, item difficulty index, and reliability and validity coefficients of the reordering of items within a power test containing ten letter-series-completion items. The results suggest that effects on test statistics from item rearrangement are, generally, minimal. The implication of these findings for test designs involving an item sampling procedure is that performance on an item is minimally influenced by the context in which it occurs.

Item Arrangement and Bias in an Interest Inventory

Item Arrangement and Bias in an Interest Inventory

Effect of test-retest interval and item arrangement on the shortened forms of the MMPI.

Effect of test-retest interval and item arrangement on the shortened forms of the MMPI.