Characteristics Of Test Items Research Articles

Differential Item Performance in Mathematics Achievement Test Items: Effect of Item Arrangement

Past research on gender differences in performance on mathematics achievement test items has focused on characteristics of test items on which males and females perform differentially. In most instances, such studies have employed a post-hoc content analyses of test items identified via statistical analyses to function differentially for males and females. This study, in contrast, used an experimental design approach to investigate the effect of item context on differential item performance on mathematics test items between males and females. The analyses were based on a modified one parameter IRT methodology. Although some significant differences in performance were found between and within item arrangements for males and females, given the number of statistical tests employed, these significances could be due to chance alone.

FINAL REPORT: EXPLOITING COLLATERAL INFORMATION IN THE ESTIMATION OF ITEM PARAMETERS

ABSTRACTWhen using item response theory (IRT) models in educational and psychological measurement, it is standard practice to estimate the operating characteristics of test items from examinees' item responses alone. This is the final report of a project that employed Bayesian and empirical Bayesian methods to exploit additional information that is often available about test items (e.g., format, content, or cognitive processing requirements) or about examinees (e.g., educational background or demographic status). Practical and theoretical results obtained in a series of research reports are summarized.

Problems Related to the Use of Conventional and Item Response Theory Equating Methods in Less Than Optimal Circumstances

This paper focuses on a discussion of how various equating methods are affected by (1) sampling error, (2) sample characteristics, and (3) characteristics of anchor test items. Studies that examine the effect of analytic techniques for smoothing or modeling mar ginal and bivariate frequency distributions on the ac curacy of equipercentile equating are reviewed. A need for simulation and empirical studies designed to evaluate the effectiveness of analytic smoothing tech niques for recovering the underlying distribution when sample size, test length, and distributional shape are varied is identified. Studies that examine the question of whether an equating transformation remains the same regardless of the group used to define it are also reviewed. The results of some studies suggested that this may not be a problem for forms of a homogene ous test constructed to be similar in all respects. Re sults of other studies indicated that examinees who take a test on different administration dates may vary in systematic ways and thus affect equating results. Finally, studies which examine the characteristics of anchor test items are reviewed. It is concluded that whenever groups differ in level and dispersion of abil ity, special care must be taken to assure that the an chor test is a miniature of the total test.

RELATIONSHIPS OF TEST ITEM CHARACTERISTICS TO TEST PREPARATION/TEST PRACTICE EFFECTS: A QUANTITATIVE SUMMARY

ABSTRACTTen studies of the effects of test preparation or test practice were located that provided effect estimates for individual test item types. These item‐level effects were related to a number of test item characteristics, including item format, difficulty level, and complexity of test directions. Strong relationships were noted between test preparation/practice effects and (a) the length and complexity of test directions and (b) the item format employed. Suggestions are made for further research on the relationship of test item characteristics to the susceptibility of tests to coaching and practice.

Relations of test item characteristics to test preparation/test practice effects: A quantitative summary.

Relations of test item characteristics to test preparation/test practice effects: A quantitative summary.

Relations of test item characteristics to test preparation/test practice effects: A quantitative summary.

Relations of test item characteristics to test preparation/test practice effects: A quantitative summary.

Structured personality test item characteristics and validity

Using the structured personality test item as the unit of analysis, the purpose of this research was to evaluate the relationship between validity and a variety of other test item parameters. Of particular interest was the relationship of test item criterion validity to negative keying and to negative wording. By drawing a distinction between negative keying and negative wording it was demonstrated that the use of balanced scales to control acquiescence need not result in a reduction in item criterion validity. Whereas the use of negative wording has in the past reduced validity, data demonstrated that positively worded, negatively keyed items did not. In addition, results indicated that clear, moderately short, relevant test items tended to be the most empirically valid.

Subjective Judgment of Multiple-Choice Item Characteristics

This study was concerned with the reliability and validity of subjective judgments about five characteristics of multiple-choice test items from an introductory astronomy test: (a) item difficulty, (b) language complexity, (c) content importance or relevance, (d) response set convergence, and (e) process complexity. Judgments of item difficulty were the least reliable. Judgments on all characteristics were significantly associated with empirical values of objective item difficulty, which served as the criterion measure. Complexity exhibited the closest relationship with both empirical (objective) and judged (subjective) item difficulty.

An Iterative Structured Judgment Process for Establishing Standards on Competency Tests: Theory and Application

This paper reports the results of research on a judgmental method for establishing standards on competency tests. The research had several objectives: first, to demonstrate the feasibility of a new, and somewhat more complex standardsetting method than has been used in the past; second, to examine the degree of similarity of standards recommended by judges having different types of interest and involvement in secondary education; third, to explore relationships between the background characteristics of judges and their recommended standards; fourth, to search for relationships between the characteristics of test items, and judges' assessments of their importance in contributing to decisions on whether students should

INTERACTIONS BETWEEN ITEM CONTENT AND GROUP MEMBERSHIP ON ACHIEVEMENT TEST ITEMS

It is often argued that certain types of items are biased against some groups. Of particular concern is the possibility that non-essential characteristics of particular test items may result in misleadingly poor performance for minority and/or socioeconomically disadvantaged children. For example, when vocabulary is incidental to the skill that the items are purported to measure (e.g., an arithmetic story problem), then the use of words that are less familiar to members of one group than to another may result in a biased indication of the relative performance of the two groups. A variety of student characteristics could interact with item characteristics to affect overall performance on a test. Ethnic group membership or socioeconomic status are but two of many potentially important characteristics. Differences in motivation or in test taking anxiety could also lead to interactions with characteristics of test items. The identification and understanding of possible interactions between student characteristics and the characteristics of items used to measure student achievement could contribute to the development of improved measurement procedures. Characteristics which have the potential to distort test results by interacting undesirably with student characteristics can be investigated experimentally. Item characteristics would be systematically varied and the performance of groups differing in characteristics such as socioeconomic status or level of anxiety would be compared experimentally. A study by Medley and Quirk (1974) is an example of this approach. Medley and Quirk used altered content specifications for the general education items of the National Teacher Examinations (NTE). For two experimental forms of the examination, the proportion of items involving contemporary culture (modern items) and the proportion of items involving black cultural contributions (black items) were increased and the proportion of items dealing with classical contributions (traditional items) was reduced. The relative performance of black and white candidates was then compared on the three types of items. The black candidates did relatively better on the black and modern items than on traditional items, and, when black and modern items were compared, their relative performance was better on the black items.

On Using the Shewhart Control Chart To Display Item Analysis Data

A procedure was described by Rogers (1978) for using statistical control charts to display test item analysis data. Rogers' charts in volved plotting individual item difficulty indices with control limits and centerline values derived from an implicitly assumed mean item difficulty of 50%. The author of the present paper contends that a more useful representation of test item characteristics can be ob tained by adopting the usual control charting convention of allow ing the data itself to determine limit and centerline values. Then these values can be compared to any criterion difficulty level(s) deemed appropriate for a given set of test items. Although control charts cannot always be applied to item analysis, the author de scribes two specific situations where they can be used.

Developing national attainment measures: A study of irish experience

Recent experience of test development in Ireland, a country which had no tradition of formal standardised testing, is drawn on, in order to highlight a number of practical and measurement problems that future test developers in other countries may experience. The paper focuses on a number of issues that have received relatively little attention in the literature. The practical problems which are examined include the identification of the objectives of the school curriculum, the production of test materials and the selection of appropriate samples of schools for standardisation purposes. The measurement or psychometric problems relate to curricular and statistical characteristics of individual test items, a test ceiling effect caused by some pupils answering all or almost all of the test items correctly and the lack of test sophistication among pupils and teachers. It is pointed out that a number of conventional procedures for developing norm-referenced measures of pupil attainment may prove somewhat less than successful in countries or educational systems with common curricula.