Achievement Test Items Research Articles

Sensitivity of Item Difficulties to Curricular Validity

The issue of the match of what is taught to what is tested recently has received increased emphasis. Many recent studies have examined the extent of the mismatch between a local set of objectives/instruction and the content of a nationally standardized test. Phillips and Mehrens (1985, 1987; see also Mehrens & Phillips, 1986) have conducted a series of studies that show that, within a district, different textbook series and informal curricula generally have no significant impact on test total, objective, or item scores. The present study explored further the curricular validity differences across textbooks and the relationship between item statistics and measures of curricular validity. In particular, this study sought to determine whether item p‐values appear to be related to measures of curricular validity based on three mathematics textbook series used at a given grade and in a previous grade. The results of this study indicated that the textbooks differed somewhat in content coverage when using a 180‐cell matrix classification. However, these differences were not great, especially when the textbooks in both grades 5 and 6 were considered. Further, all three series covered almost all of the 53 cells in the matrix covered by the Stanford Achievement Test, and the differences that did exist in textbook content coverage had no observable relationship to differences in item p‐values. In addition, the mean difficulty level of the Stanford Achievement Test items classified by cell were similar for students using the different textbooks, despite the differences among textbooks in location, presentation, and organization of content.

Item-Difficulty Sequence in Achievement Examinations: Examinees' Preferences and Test-Taking Strategies

Elementary, secondary school, and university students were asked for the test-taking strategy usually used in taking an achievement test and for the sequence of difficulty preferred for achievement test items. The majority preferred an easy-to-hard strategy with items in order-presented as second choice, and the hard-to-easy strategy chosen least often. No conclusions were drawn as to the effect of students' usual test-taking strategies on research with various item-difficulty sequences. The data on students' preferences for item sequences support the use of the easy-to-hard or random arrangement over the hard-to-easy.

The number line as a teaching aid

The number line is often recommended as an aid for the teaching of whole number addition and subtraction. The possible stages in young children's use of the aid are described, as is their ultimate outgrowing of this aid. Number line diagrams are also used in diagnostic and achievement test items to assess the understanding of whole number addition. This practice is discussed and criticised with the aid of the empirical findings of the second NAEP. Finally the wider role of number line diagrams in mathematics teaching is outlined.

Learning Disabled Students' Spontaneous Use of Test-Taking Skills on Reading Achievement Tests

The present investigation was undertaken to identify the type of strategies learning disabled (LD) students employ on standardized, group-administered achievement test items. Of particular interest was level of strategy effectiveness and possible differences in strategy use between LD and nondisabled students. Students attending resource rooms and regular third-grade classes were administered items from reading achievement tests and interviewed concerning the strategies they had employed in answering the questions and their level of confidence in each answer. Results indicated that (a) LD students were less likely to report use of appropriate strategies on inferential questions, (b) LD students were less likely to attend carefully to specific format demands, and (c) LD students reported inappropriately high levels of confidence.

The Reliability of Six Item Bias Indices

The reliabilities of six item bias indices were inves tigated for each of the eleven tests of the Iowa Tests of Basic Skills, using random samples of fifth-grade students. The reliability of an index was defined as its stability from one randomly equivalent group to an other. Both racial and sexual bias were considered. In addition, correlations among bias indices were investi gated. The results indicate that the item bias indices investigated were fairly unreliable when based on sample sizes of 200 minority and 200 majority exami nees. Consequently, this study suggests that, with sample sizes of about 200, the use of item bias indices to screen achievement test items cannot be expected to lead to consistent decisions about which items are biased.

ITEM RESPONSE PATTERNS: APPLICATIONS FOR EDUCATIONAL PRACTICE

In most uses of achievement tests, the primary focus is on a few summary scores that are based on the number of items answered correctly. The focus on the number of correct answers or some transformation of it (e.g., a grade-equivalent score) is natural. Certainly, the overall level of performance in a content area such as arithmetic is a major consideration. Furthermore, the interrelatedness of subsets of achievement test items (e.g., addition and subtraction items) makes it difficult to obtain reliable information about special strengths and weaknesses of individuals or groups of students in addition to a global summary score. In situations when the ranking of students on a global dimension is emphasized, the role of specific test content receives relatively little attention. While it may be useful to know that the average score of students in an instructional program is well below the national mean on a fourth-grade mathematics test, this information does not indicate the types of arithmetic skills the students have. As shown by Porter, Schmidt, Floden, and Freeman (1978; See also Schmidt's article in this issue), there is wide variation in content coverage of the fourth-grade mathematics tests from the four most widely used achievement test batteries. Furthermore, the level of performance can be expected to be influenced by the degree of overlap between the instructional content and the content of the test (e.g., Armbruster, Steven, & Rosenshine, 1977; Leinhardt & Seewald, 1981; Madaus, Kellaghan, Rakow, & King, 1979). Indeed, the specific match between the format of instructional exercises and the format of test questions can have a substantial impact on test scores (Alderman, Swinton, & Braswell, 1979; House, Glass, McLean, & Walker, 1978). The limitations of global summary scores can also be seen by considering their mathematical foundation. The same number-right score can be obtained in many different ways. Even on a short test of just five items, a score of three can be obtained by ten different combinations of correct and incorrect responses. The number of possible combinations of right and wrong answers expands rapidly with increases in test length. On a 10-item test, for example, there are 252 different response patterns that yield the same number-right score of five.

The episodic/semantic memory distinction as an heuristic in the study of instructional effects on cognitive structure

Previous attempts to measure the effects of instruction on students' cognitive structure have produced inconsistent and ambiguous findings. One reason may be that researchers have not distinguished between well-learned, abstracted information and memorized facts or formulas. Using the distinction between episodic and semantic memory as an heuristic, a procedure was designed to increase the likelihood that student performance on structure tasks was indicative of semantic rather than episodic memory. Significant correlations between measures of cognitive structure and performance on achievement test items were observed. In addition, partial correlations revealed that the measures of cognitive structure are correlated with the achievement items tapping higher-order cognitive processes and not with items tapping lower-order processes. These results are interpreted as supporting the argument that related research needs to be guided by theory-based definitions of cognitive structure.

Using Results on K Out of N System Reliability to Study and Characterize Tests'

For a specific achievement test item and a randomly selected examinee, let p be the probability of correctly determining whether the examinee knows the correct response. Various techniques have been proposed for estimating p. This paper describes and illustrates how results in the engineering literature on "k out of n system reliability" can be used to study and characterize tests based on the estimated values of p. In particular, we can empirically determine the minimum number of distractors required for multiple-choice tests. If we estimate p with an answer-until-correct scoring procedure, we can also determine the minimum number of examinees needed to be reasonably certain about whether y is less than or greater than some predetermined constant, where y = [UNKNOWN] pi and pi is the value of p for the ith item on an n-item test. In other words, we can determine whether the expected number of correct decisions on an n-item test is reasonably large.

A LOOK AT CONTENT BIAS IN IQ TESTS

Sexual and racial biases persist even in the face of legal sanctions. While the past century has seen females and nonwhites enfranchised, their equality has yet to be realized. Pejorative stereotypes abound in present-day society, fostering and maintaining discriminatory attitudes. While the more obvious forms of discrimination have been outlawed by the Civil Rights Act of 1964 and by Title IX of the Educational Amendment Act of 1972, more subtle forms of discrimination have yet to be eradicated. In the arena of educational assessment, one potential form of subtle bias may exist in the content of IQ test items. Although research has substantiated the existence of bias in achievement test item content (Donlon, Ekstrom, Lockheed & Harris, 1977; Tittle, McCarthy & Steckler, 1974) and performance differences (Donlon et al., 1977; Strassberg-Rosenberg & Donlon, 1975), research on IQ test bias has been limited to performance (Jencks, Smith, Acland, Bare, Cohen, Gintis, Heyns & Michelson, 1972; Jensen, 1971; Laosa, 1977; Samuda, 1975). Perhaps a reason that so much more research has been done on bias in test performance than in test item content relates to the relative ease with which performance bias can be detected. Disparities in mean IQ scores are straightforward. However, differences in the ways which stereotypes are depicted in test items require clear definitions of a biased representation. In addition to assessing the extent to which individual items are stereotyped, an analysis of content bias in IQ tests requires measurement of the frequency with which members of different races and sexes appear. Although stereotyping can be readily identified on an item by item basis, the cumulative effects of disproportionate sexual and racial representations are not apparent until the test is considered in its entirety. Regardless of whether item content bias affects subsequent test performance, the issues of sexual and racial stereotyping and disproportionate representations warrant scrutiny in and of themselves. Just as blatant forms of bias have no place in education, more subtle forms are equally intolerable even if they do not in turn effect performance. To the extent that test items are biased, they foster inappropriate and inaccurate perceptions. Considering the widespread use and respectability of IQ tests, it becomes important to ask: To what extent do test items accurately represent the distribution of sexes and races, and to what extent are members of these groups stereotyped?

Sex Bias in Achievement Test Items: Do They Have any Effect on Performance?

We may be inadvertently giving some advantage to the students of one sex in multi-quiz courses using objective items.

PARTITIONING METHODS FOR DETECTING MISCONCEPTIONS OF CONTENT AND TEST ITEMS

A variety of empirical approaches have been used for studying test items, ranging from item discrimination indices to latent trait methods (for example, Davis, 1951; Lord & Novick, 1968). Typically, such approaches have relied on data from answers to test items. The empirical data for the present study, however, were based on sortings of the items themselves, where each respondent partitioned the items according to his or her own perceptions. These sortings were analyzed using the procedures of target partition analysis (Pruzek & Pfeiffer, 1972; Pruzek, Stegman & Pfeiffer, 1972) and hierarchical clustering analysis (Hartigan, 1972, 1975; Johnson, 1967). In this study, achievement test items were used as stimuli to be sorted by groups of students having different levels of sophistication with the content. One purpose of the study was to evaluate the methodology to study how items are perceived by students and to determine how students organize content. Another purpose was to determine whether sortings by members of those groups with greater sophistication would agree more with target sortings than would sortings by members of groups with less sophistication.

Curriculum Biases in Reading Achievement Tests

The extent and direction of curriculum bias in standardized reading achievement tests are examined. Bias is estimated by comparing the relative overlap in the contents of five separate reading achievement tests with the content of seven commercial reading series at first and second grade levels. Overlap between each achievement test and each reading series is reported in terms of achievement test grade equivalent scores that would be expected given mastery of the words which appear both as content in a reading series and as achievement test items. Results indicate clear discrepancies between the grade equivalent scores obtained both between tests for a single curriculum and on a single test for different reading curricula. The implications of the apparent curriculum bias of achievement tests are discussed as they relate to teacher, child, and curriculum evaluation, to reading replacement, and to applied educational research.

A Comparison of Objective-Based and Modified-Bormuth Item Writing Techniques

Two techniques for writing achievement test items to accompany instructional materials were contrasted: (a) writing items from statements of instructional objectives, and (b) writing items from semi-automated rules for transforming instructional statements (adapted from Bormuth). Items of each type were written by two experienced item writers. Students were given tests employing these items before and after reading a programmed booklet. One item writer was found to produce consistently easier items than the other regardless of the item writing technique employed. Both item writing techniques resulted in about the same number of faulty items, indicating that the "subjectivity" found in traditional item writing was also present with the semi- automated techniques.

Standardized Achievement Tests: How Useful for Special Education?

The extent and direction of curriculum bias in standardized reading achievement tests are examined. Bias was estimated by comparing the relative overlap in the contents of four separate reading achievement tests with the contents of five commercial reading series at first and second grade levels. Overlap between each achievement test and each reading series is reported in terms of achievement test grade equivalent scores that would be expected given mastery of the words that appear both as content in a reading series and as achievement test items. Results indicate clear discrepancies between the grade equivalent scores obtained, both between tests for a single curriculum and on a single test for different reading curricula. The implications of the apparent curriculum bias of achievement tests are discussed as they relate to evaluation of teachers, children, and curricula; to reading placement; and to the identification and classification of exceptional children.

SOME CORRELATES OF NET GAIN RESULTANT FROM ANSWER CHANGING ON OBJECTIVE ACHIEVEMENT TEST ITEMS

Few teachers include test-taking strategies as part of their curricula. Nevertheless, students do acquire strategies to use against their recurring enemy-the objective achievement test. One tactic of test-taking strategy deals with answer changing behavior; whether to change a response after deliberation, or not to change it, is the student's dilemma. Many students do change answers, but research indicates that students generally believe that changing answers is an unwise tactic (Foote & Belinky, 1972; Mathews, 1929). In a graduate level educational measurement course, the authors found that 64 percent of the students believed that changing answers would tend to lower a student's total score; 36 percent believed that changing answers would neither raise nor lower total score. None believed that changing answers would increase scores. Contrary to students' expectations, research studies report that most students gain in total score by changing answers (Bath, 1967; Jacobs, 1972; Reiling & Taylor, 1972). A small percentage of the students in the above mentioned studies did, however, decrease their total scores as a consequence of changing answers. The purpose of this study was to determine the relationship of sex, answer-changing incidence, and total score to net changes in total score resulting from changing answers, by examining the answer-changing behavior of a large number of graduate students responding to achievement test items.

An Assessment of the Effectiveness of Complex Alternatives in Multiple Choice Achievement Test Items

Discrimination indices and difficulty levels of multiple-choice achievement test items containing only substantive response alternatives were compared with items containing each of three complex alternative types: All of the above; None of the above; and combination complex alternatives (e.g., A and B; Either A or B; Two of the above, etc.). The same item statistics were compared, with items reclassified according to type of alternative keyed as the correct answer. Items containing combination complex alternatives were found to be most difficult, and items containing an "All of the above" alternative were found to be least difficult (especially when that alternative was keyed as the correct answer). Discrimination index was less affected by the inclusion or exclusion of complex alternatives than was difficulty level, but the highest discrimination occurred in items containing only substantive alternatives; the lowest in items in which "None of the above" was keyed as the correct answer. It was also found that all three types of complex alternatives functioned better as distractors than did substantive alternatives, with combination complex distractors receiving the highest rate of response.

On the Theory of Achievement Test Items. John R. Bormuth

Previous articleNext article No AccessBook ReviewsOn the Theory of Achievement Test Items. John R. Bormuth Richard C. AndersonRichard C. Anderson Search for more articles by this author PDFPDF PLUS Add to favoritesDownload CitationTrack CitationsPermissionsReprints Share onFacebookTwitterLinkedInRedditEmail SectionsMoreDetailsFiguresReferencesCited by Volume 80, Number 4Aug., 1972 Article DOIhttps://doi.org/10.1086/443056 Views: 1Total views on this site Journal History This article was published in The School Review (1893-1979), which is continued by the American Journal of Education (1979-present). Copyright 1972 The University of ChicagoPDF download Crossref reports no articles citing this article.

Book Reviews : John R. Bormuth. On the Theory of Achievement Test Items, with an Appendix by Peter Menzel, On the Linguistic Bases of the Theory of Writing Items. Chicago: University of Chicago Press, 1970. Pp. x + 163. $9.75

Book Reviews : John R. Bormuth. On the Theory of Achievement Test Items, with an Appendix by Peter Menzel, On the Linguistic Bases of the Theory of Writing Items. Chicago: University of Chicago Press, 1970. Pp. x + 163. $9.75

An Investigation of the Effects of “Verbal Load” in Achievement Tests

Now, as never before, educational and psychological tests are being examined for possible bias against disadvantaged persons or members of minority groups in the United States. This is a study of one possible source of bias, viz., the language of multiple choice achievement tests. It can be argued that the language used in multiple choice achievement test items should be no more complex than is necessary to test the examinee's knowledge of the subject matter. Language complexity above this minimum level can be regarded as verbal overload and may constitute a source of bias against those people whose verbal skills are limited. Achievement tests with verbal overload might have any or all of the following undesirable consequences:

AN INVESTIGATION OF SOME OF THE EFFECTS OF “VERBAL LOAD” IN ACHIEVEMENT TESTS

ABSTRACTIt was argued that the language used in multiple choice achievement test items should be no more complex than is necessary to test the examinee's knowledge of the subject matter. Excessive complexity might constitute a source of bias against people with limited verbal skills. A test of this hypothesis was made by comparing a simplified language version of STEP social studies test items against an “original language form” with junior and senior high school students in Oakland, California. About two‐thirds of the students were Negro. Mean reading comprehension scores fell at about the 30th percentile on national norms. Test results on both forms appeared to be internally consistent and in accord with general expectations for this student sample. There was, however, no support for the notion that the simplified language test version would yield different results. Test difficulty and relationships with reading comprehension scores and social studies grades were essentially the same for both forms.