Reading Comprehension Items Research Articles

Factors Affecting the Difficulty of Reading Comprehension Items for Successful and Unsuccessful Readers

This study assessed the contributions of various test features (passage variables, question types, and format variables) to reading comprehension performance for successful and unsuccessful readers. Items from a typical standardized reading comprehension test were analyzed according to 20 predictor test features. A three-stage conditional regression approach assessed the predictability of these features on item-difficulty scores for the two reader groups. Two features, location of response information and stem length, accounted for a significant amount of explained variance for both groups. Possible explanatory hypotheses are considered and implications are drawn for improved test design as well as for further research concerning interactions between assessment task features and reader performance.

THE INTERNAL CONSTRUCT VALIDITY OF THE SAT1

ABSTRACTThe primary goal of this research was to obtain a fuller understanding of what is being measured by the SAT. In addition, this project had two secondary goals: (1) the identification of a set of analytic techniques that can satisfactorily assess the dimensionality of the SAT; and (2) the identification of components of a cost‐effective and informative system for monitoring the internal construct validity of the SAT.The principal project findings are: The SAT‐Mathematical test is effectively unidimensional. There is little empirical justification for dividing the total SAT‐Mathematical score into subscores on the basis of item content related to subject area. For purposes of differential item functioning analysis, the total SAT‐Mathematics score should be an adequate matching variable. The SAT‐Verbal test is composed of two distinct albeit highly‐related dimensions, a reading dimension and a vocabulary dimension. Hence, there is an empirical justification for reporting two separate subscores. Currently, antonym and analogy items count toward a vocabulary subscore; sentence completion and reading comprehension items count toward a reading subscore. The empirical data indicate that the sentence completion items appear to belong more with the analogy and antonym items than with the reading comprehension items. The LISREL analyses also provide some evidence for separating the Vocabulary dimension into two dimensions, Antonyms and Analogies. The evidence is inconsistent, however. For the purposes of differential item functioning analysis, it may be necessary to match on the current Reading subscore (reading and sentence completion) for reading comprehension items, an analogy/sentence completion criterion for analogies, an antonym/sentence completion criterion for antonyms and any of these three criteria or SAT‐Verbal for sentence completion items. This research has several findings that have implications for redesigning the SAT‐Verbal test. First, the sentence completion item type is the item type that is most like the total SAT‐Verbal score in what it measures, while the reading passage items are measuring the construct that is least like the total SAT‐Verbal score. Second, speededness effects on SAT‐Verbal may be large enough to produce a factor in item level data that is more salient than a general reading passage factor. Third, the analogy item seems to be the least reliable item type, while the sentence completion item type appears to be the most reliable. This array of results points in the direction of using more reading passage items to achieve a more distinct and reliable reading subscore, using more sentence completion items because they are more reliable and may take less time than other items, and using fewer analogy items because they are less reliable and probably more time consuming than sentence completions or antonyms. The analyses conducted in this study have served to underscore the value of using confirmatory factor analysis of item parcel data to study the dimensional structure of test data. This approach is computationally inexpensive, and appears to provide meaningful and consistent results. The use of parallel parcels makes item data amenable to linear factor analysis. The method seems to circumvent the problems associated with directly factor analyzing item data, namely, the propagation of artifactual “difficulty” factors. The method can also avoid the problem of observing a “speed” factor, as items from later positions in the test can be balanced across parcels. In sum, the parallel parcel approach can be used to dispense with difficulty and speed factors and, hence, obtain a clearer look at the substantive factor structure of the test. Full information factor analysis is not ready for routine use with ATP data. The TESTFACT program is very expensive and sometimes yields “methodological” factors under the full information mode. The TESTFACT program can also produce a least squares factor analysis of a smoothed positive definite matrix of tetrachorics corrected for guessing at a fraction of the cost of the full information solution. Unfortunately, this solution appeared to yield difficulty factors for both SAT‐Verbal and SAT‐Mathematical.

A COMPARISON OF FREE‐RESPONSE AND MULTIPLE‐CHOICE QUESTIONS IN THE ASSESSMENT OF READING COMPREHENSION

ABSTRACTUndergraduate students completed a Reading Comprehension test in which each of a number of passages served as the basis for four to eight questions, half framed in multiple‐choice format and half requiring that the examinee produce an appropriate answer. Questions within a passage were balanced according to the kind of information processing they were thought to require: Explicit questions dealt with information given explicitly in the passage; Inference questions required inference based on material presented; and Application and Evaluation questions required that the examinee go beyond what was presented to consider applications of ideas presented or evaluation of the logic or the style of the passage. It was hypothesized that questions of the first type would show no systematic differences associated with the format of the question, while those of the second might, and those of the third would, draw on somewhat different cognitive abilities when examinees were required to produce rather than merely to recognize an appropriate answer.Correlational and factor analyses provided only minimal indications of systematic differences associated with the response format. Estimated true correlations across formats averaged .91, a relation of the same magnitude as that found when multiple‐choice Reading Comprehension items differing in their information processing requirements are contrasted with one another. Weak evidence for a factor loaded by scores based on performance on multiple‐choice but not free‐response items was found, but only one score had a loading as great as .3 on this factor.Relations of test scores with other cognitive measures also failed to show systematic differences associated with format. Regardless of differences in item type and response format, all scores based on the Reading Comprehension test showed fairly substantial relations with tests of Reasoning and Vocabulary and with the ACT English achievement test, moderate relations with the ACT mathematics achievement score, and near‐zero relations with a test of Divergent Thinking.Despite this evidence for similarity in performance across formats, however, there is also evidence of systematic differences for individuals. The subjects by format interaction in the analysis of variance, while not of great strength, was significant at the five percent level of confidence. This effect indicates that some students showed larger differences in performance between the two response formats than did others.It was speculated that two changes in test items might be required to elicit the differences in performance associated with response format that have been observed in some other testing situations. Items may need to be structured so that they draw less on information given in or derivable from the test materials and more from information whose relevance is not specified for the examinee; and types of questions with which examinees are not so well practiced may need to be employed.

Internal Construct Validity of a College Admissions Test Across Handicapped and Nonhandicapped Groups

This study investigated the comparability of SAT Verbal and Mathematical scores for nonhandicapped examinees taking standard administrations and handicapped populations taking extended-time, special administrations. A simple two-factor confirmatory model based on Verbal and Mathematical item parcels was posed and tested for invariance across two sets of handicapped and nonhandicapped samples, with each set taking a different form of the SAT. Results showed the two common-factor model to provide a reasonable fit in all populations with the mathematical factor generally fitting somewhat better than the verbal one. Compared with the nonhandicapped population, the factors tended to be less correlated in most of the handicapped groups. In addition, some evidence of verbal method factors defined by the antonym and reading comprehension item types was evident. Finally, there was some suggestion that one of the two Mathematical item types led to different observed score scale units for learning disabled students taking a cassette administration.

CONTRIBUTION OF VERBAL ITEM TYPES IN THE GRE GENERAL TEST TO ACCURACY OF MEASUREMENT OF THE VERBAL SCORES

ABSTRACTThe main purpose of this study is to compare the contribution to measurement accuracy of the verbal score of each of the four verbal item types included in the GRE General Test. Comparisons are based on item response theory, a methodology that allows the researcher to look at the accuracy of individual points on the score scale. This methodology is based on the assumption that the four verbal item types measure the same verbal ability. Since the results of the study do indicate that the reading comprehension item type measures something slightly different from what is measured by sentence completion, analogy, or antonym item types, only tentative conclusions may be drawn.The antonym item type contributes the most accuracy of the four item types for scores above about 550. Analogy items contribute to the measurement accuracy of verbal ability throughout the score range. This is especially true when item types are matched on verbal difficulty. These results suggest that the analogy and antonym item types are useful for maintaining accuracy of the verbal score scale at the upper levels. Eliminating these items might have a serious impact on the validity of the GRE verbal score in the upper regions of the scale. Studies of the validity of item types at the upper score range using external criteria would be necessary to understand the exact contribution of the item types to the validity of the test.

THE INTERNAL CONSTRUCT VALIDITY OF THE SAT ACROSS HANDICAPPED AND NONHANDICAPPED POPULATIONS

ABSTRACTThis study investigated the comparability of SAT Verbal and Mathematical scores for one nonhandicapped and nine handicapped populations. A simple two‐factor model based on Verbal and Mathematical item parcels was posed and tested for invariance across populations with respect to: (1) the number and intercorrelation of factors, (2) the pattern of factor loadings, and (3) the equality of scale units.The two common‐factor model provided a reasonable fit in all populations, with the mathematical reasoning factor generally showing a better fit to the population data than the verbal reasoning factor. Compared with the nonhandicapped population, these factors tended to be less correlated in most of the handicapped groups. This somewhat greater specificity implies the increased likelihood of achievement growth in one area independent of the other, suggesting that the two scores be interpreted separately rather than as an SAT composite.With respect to the pattern of factor loadings, some indication of the presence of two verbal method factors was discovered. Antonym items and, to a lesser extent, reading comprehension items formed such factors. While present to a certain extent for all groups, the method variance seemed to be somewhat greater for the learning disabled populations.Finally, there was evidence that multiple‐choice Mathematical items led to different observed score scale units for the learning disabled students taking a cassette administration. Because of the small sample sizes on which this finding is based, it should be treated with caution. However, if dependable, the finding suggests that Mathematical scores may underestimate the reasoning ability of these students.

Measuring Passage Contribution in ESL Reading Comprehension

Reading comprehension items are valid to the extent that they measure what subjects have understood of the stimulus material. This article reports an empirical analysis of two administrations of two reading tests: the first time, without the reading passages, and the second time, with the passages. Data from the two administrations were used to calculate the passage dependency of each test, that is, the extent to which questions can be answered without reading the texts upon which the questions are based. The two tests in this research, Davis Reading Test (Davis and Davis 1956) and Cooperative English Tests (Educational Testing Service 1960), exhibited little passage dependency. The stability of item types across the two presentation conditions is discussed, and a hierarchy of item-type difficulty is established using latent trait measurement logits of difficulty. The results of this study suggest that classroom teachers should examine commercially available tests carefully for passage dependency. Furthermore, in constructing reading tests, teachers should avoid writing items that test general knowledge. Instead, teachers should strive to write items that test memory organization and that reveal whether or not inferences have been drawn.

Comparative analyses of English as a Second Language reading comprehension data

Item responses on a multiple-choice reading comprehension test from 88 ESL subjects were submitted to classical test item analyses and the Rasch one-parameter latent trait model. The Rasch model identified more misfitting items than the classical item analyses and allowed the researchers to plot the reading comprehension items at their calibrated positions along the lines of the latent trait. The trait was defined at the easy end by items that assess literal comprehension and inference of explicit details and at the difficult end by items that assess multiple string inferences. The Rasch model also indicated that additional items were needed at both ends of the difficulty continuum, if subjects in those ranges are to be discriminated properly.

Passage Independence in Reading Achievement Tests: A Follow-up

38 college undergraduates were administered reading-comprehension items from a major standardized achievement test with corresponding passages deleted. Analysis indicated that, after 20 years of similar research findings, highly passage-independent items still occur on major tests.

DETECTION OF A SKILL DICHOTOMY USING STANDARDIZED ACHIEVEMENT TEST ITEMS

Multiple‐choice reading comprehension items from a conventional, norm‐referenced reading comprehension test are successfully analyzed using a simple latent class model. A classification rule for assigning respondents to “mastery” or “nonmastery” states is presented which simplifies the scoring procedure of Macready and Dayton (1977). A procedure is also derived for estimating the “true,” or “disattenuated,” latent cross‐classification of masters versus nonmasters for two tests, and illustrated using two sets of items from the same content domain. Results support the use of latent class, state mastery models with more heterogeneous item pools than has been advocated by previous authors.

CONVERGING ON CORRECT ANSWERS: A PECULIARITY OF MULTIPLE CHOICE ITEMS

The primary mode of mental measurement in the United States today is the multiple choice item. The multiple choice format has many commendable qualities: it is relatively straightforward in terms of what is required from the subject, it is well-understood by most test-takers, it affords highly objective scoring, and it is computer scannable. The format does have some drawbacks: it is somewhat limiting in the type of mental behaviors which can be elicited, it is subject to guessing, and it interacts with characteristics of test-takers. It is this final aspect that is the subject of this research. There are certain aspects of multiple choice items which are consistent across the traits or content areas being measured. The basic form of a multiple choice item includes: (1) a set of data pertinent to the item, (2) a question or task which must be responded to, and (3) a set of alternative answers, one of which is correct. Some multiple chdice items require the test-taker to solve a problem and then locate his answer among the distractors (such as mathematics and some abstract reasoning items). Other multiple choice items require the test-taker to compare and contrast the alternative responses and pick the best one (such as reading comprehension and sentence completion items). For some items either approach may be used. An unstated assumption of multiple choice testing is that all test-takers are equally adept at responding to problems presented in the fashion just described. If aspects of the multiple choice format interact with characteristics of test-takers, then the resultant measures will be confounded to some degree. Specifically, some individuals may find the multiple choice format confusing while others find that it provides inadvertant clues to correct answers. As a result, scores are deflated or inflated due to an ability which is logically independent of the trait being measured. When the interaction between the characteristics of an item and a person is viewed from the perspective of the item, it is generally considered to be an issue of item construction; when viewed from the perspective of the person, it is considered to be an issue of testwiseness. Although there have been a number of interesting and potentially useful advances in item writing over the past decade (e.g., Bormuth, 1970 and Guttman, 1970, echoed more recently by Wood, 1977 and Nunnally, 1978), the contention of Wesman (1971) that item writing is still an art holds true in general. If item writing has proven somewhat resistant to being turned into a science, it is highly restrictive as an art form. Most texts on test construction provide a list of prescriptions and proscriptions for budding item writers. One major purpose of these lists is to keep the item writer from inadvertantly cueing the test-taker as to which of the choices is correct. There are a variety of ways in which the correct answer might be revealed to the adept test-taker. Gronlund (1977) has an excellent list of pitfalls for those who are not familiar with them.

THE FEASIBILITY OF USING ITEM RESPONSE THEORY AS A PSYCHOMETRIC MODEL FOR THE GRE APTITUDE TEST

THE FEASIBILITY OF USING ITEM RESPONSE THEORY AS A PSYCHOMETRIC MODEL FOR THE GRE APTITUDE TEST

THE EFFECT OF THE POSITION OF AN ITEM WITHIN A TEST ON ITEM RESPONDING BEHAVIOR: AN ANALYSIS BASED ON ITEM RESPONSE THEORY

ABSTRACTThe research described in this paper deals solely with the effect of the position of an item within a test on examinee's responding behavior at the item level. For simplicity's sake, this effect will be referred to as practice effect when the result is improved examinee performance and as fatigue effect when the result is poorer examinee performance. Item response theory item statistics were used to assess position effects because, unlike traditional item statistics, they are sample invariant. In addition, the use of item response theory statistics allows one to make a reasonable adjustment for speededness, which is important when, as in this research, the same item administered in different positions is likely to be affected differently by speededness, depending upon its location in the test.Five types of analyses were performed as part of this research. The first three types involved analyses of differences between the two estimations of item difficulty (b), item discrimination (a), and pseudoguessing (c) parameters. The fourth type was an analysis of the differences between equatings based on items calibrated when administered in the operational section and equatings based on items calibrated when administered in section V. Finally, an analysis of the regression of the difference between b's on item position within the operational section was conducted. The analysis of estimated item difficulty parameters showed a strong practice effect for analysis of explanations and logical diagrams items and a moderate fatigue effect for reading comprehension items. Analysis of other estimated item parameters, a and c, produced no consistent results for the two test forms analyzed.Analysis of the difference between equatings for Form 3CGR1 reflected the differences between estimated b's found for the verbal, quantitative, and analytical item types. A large practice effect was evident for the analytical section, a small practice effect, probably due to capitalization on chance, was found for the quantitative section, and no effect was found for the verbal section.Analysis of the regression of the difference between b's on item position within the operational section for analysis of explanations items showed a rather consistent relationship for Form ZGR1 and a weaker but still definite relationship for Form 3CGR1.The results of this research strongly suggest one particularly important implication for equating. If an item type exhibits a within‐test context effect, any equating method, e.g., IRT based equating, that uses item data either directly or as part of an equating section score should provide for administration of the items in the same position in the old and new forms. Although a within‐test context effect might have a negligible influence on a single equating, a chain of such equatings might drift because of the systematic bias.

Components of Answers to Multiple-Choice Questions on a Published Reading Comprehension Test: An Application of the Hanna-Oaster Approach

A PERSISTENT PROBLEM in reading comprehension tests is that factors other than comprehension of the material itself can influence answers to the questions. The Hanna-Oaster approach to assessing context dependence, based on both context independence and context dependence indices, was applied to data from a published reading comprehension test. The results are discussed both in terms of item statistics and passage-level statistics. The data indicated that for many items, the context independence index was much higher than is considered desirable in the Hanna-Oaster formulation. At the passage level, this index correlated negatively with estimates of passage difficulty derived from readability indices and from cloze scores. The context dependence index, on the other hand, correlated positively with both the cloze and readability indices. Some implications of the Hanna-Oaster approach for analysis of reading comprehension items, as well as its potential use for schema and context theorists, are suggested.

A Psycholinguistic Approach to Training Urban High School Students in Test-Taking Strategies

Test-wiseness is an important concept of psychological measurement. Research has been consistent in pointing out that test-wiseness can be learned and taught by a. variety of methods including programmed materials, lecture-supervision and reading about the test-wiseness cues. Studies conducted on Black and other minority student populations, however, have been generally inconclusive. Test-wiseness operates as error variance, and its effect is to reduce the validity and reliability of tests and to make test items easier. Many standardized reading comprehension tests have been found to be high in passage-independency, or the probability of an item being answered correctly by someone who has not read the passage. It is probable that many students possess a kind of testwisdom that enables them to score higher than chance on such faulty reading comprehension items. An attempt should be made to identify students who are low in test-wiseness and such students should be instructed in test-taking strategies. Recent notions about language development, particularly as it has been described by those who work within the generativetransformational framework, have been of great interest to researchers in the field of reading. The rejection of the passive, receptive learner for one who is actively constructing his language, the competence-performance distinction, the analysis of grammatical structure-these and other ideas have helped to lay the founda-

Passage-Dependency Data in the Selection of Reading Comprehension Test Items

The reading comprehension items for a revision of a widely used standardized reading test were administered to several hundred fourth and sixth-grade pupils in order to determine the passage dependency of each item. All of the field-tested reading comprehension items were divided into nine booklets, which contained one normal reading comprehension subtest and one subtest with questions about passages that had been deleted. An index of passage dependency was computed for each item, and the index was used to help the authors locate items that were weak. The index also indicated the effect of passage dependency on skill categories.

Objective Evaluation of the Quality of Multiple-Choice Test Items Designed to Measure Comprehension of Reading Passages

THE QUALITY OF READING COMPREHENSION test items may be determined, in part, by computing an index that indicates the extent to which correct answers to such items can be identified in the absence of the reading passages to which they refer. Data obtained in this study indicate that such an index can be determined with an adequate degree of reliability. The data also are used to illustrate how the strengths and weaknesses of particular items may be determined. Procedures that have been used to varying degrees in the past for producing reading comprehension items of high quality are discussed.