Item Selection Criteria Research Articles

Bayesian item selection criteria for adaptive testing

Owen (1975) proposed an approximate empirical Bayes procedure for item selection in computerized adaptive testing (CAT). The procedure replaces the true posterior by a normal approximation with closed-form expressions for its first two moments. This approximation was necessary to minimize the computational complexity involved in a fully Bayesian approach but is no longer necessary given the computational power currently available for adaptive testing. This paper suggests several item selection criteria for adaptive testing which are all based on the use of the true posterior. Some of the statistical properties of the ability estimator produced by these criteria are discussed and empirically characterized.

Some New Item Selection Criteria for Adaptive Testing

In this study some alternative item selection criteria for adaptive testing are proposed. These criteria take into account the uncertainty of the ability estimates. A general weighted information criterion of which the usual maximum information criterion and the proposed alternative criteria are special cases is suggested. A small simulation study was conducted to compare the different criteria. The results showed that the likelihood weighted information criterion is a good alternative to the maximum information criterion. Another good alternative is a maximum information criterion with the maximum likelihood estimator of ability replaced by the Bayesian expected a posteriori estimator.

Research Validity Scales for the NEO--PI--R: Development and Initial Validation

The purpose of this investigation was to develop a set of research validity scales for use with the NEO Personality Inventory-Revised (NEO-PI-R; Costa & McCrae, 1992b). In study 1 we used the existing NEO-PI-R item pool to select items for three validity scales: positive presentation management, negative presentation management, and inconsistency. Several iterative item and scale analyses were conducted, using multiple criteria for item selection. These analyses resulted in 10-item scales. In Study 2, the internal consistency, interscale relationships, and normative characteristics were examined in a separate sample of working adults. In Study 3, the validity of the scales was examined by contrasting five sets of NEO-PI-R protocols: from a separate sample of working adults, from a sample of 100 NEO-PI-R protocols with randomly produced responses, and from three samples of undergraduates completing the NEO-PI-R under different instructional sets. Analyses revealed that both the research validity scales and the NEO-PI-R domain scales were sensitive to group differences. issues relating to the appropriate use of the scales are discussed.

Some New Item Selection Criteria for Adaptive Testing

In this study some alternative item selection criteria for adaptive testing are proposed. These criteria take into account the uncertainty of the ability estimates. A general weighted information criterion is suggested of which the usual maximum information criterion and the suggested alternative criteria are special cases. A simulation study was conducted to compare the different criteria. The results showed that the likelihood weighted mean information criterion was a good alternative to the maximum information criterion. Another good alternative was a maximum information criterion with the maximum likelihood estimate of ability replaced by the Bayesian EAP estimate. An appendix discusses the interval information criterion for the two- and three-parameter logistic item response theory model.

Are we looking for parsimony, or what?

In a comment on the special issue of the European Journal of Personality on Factor V from the Five‐Factor Model, I argue that attempts to arrive at a definitive interpretation are premature in view of the limitations of the item pools that have been used, arbitrariness in choosing criteria for item selection, and limitations of sample sizes. At present, the best tentative interpretation of V is Creativity/Imagination. However, the relation of this concept to the domain of measured intelligence should be reconsidered. The joint study of temperament (Factors I and IV), character (II and HI), and intelligence factors should provide insight into the fascinating blends that arise between these dimensions. Such blends are captured by the Abridged Big‐Five Dimensional Circumplex approach to personality structure in a parsimonious manner.

The Application of Strength of Association Statistics to the Item Analysis of an In-Training Examination in Diagnostic Radiology

Using item responses from an intraining examination in diagnostic radiology, the application of a strength of association statistic to the general problem of item analysis is illustrated. Comparisons are made between the use of the omegasquared and the pointbiserial as criteria for item selection. General issues of reliability and errors of measurement are discussed.

Effects of Instructions and Rating Scales on Item Selection for the BSRI Scales

Walkup and Abbott (1978) stated that Edwards and Ashworth's (1977) failure to replicate Bem's (1974) selection of items for the Masculinity and Femininity Scales of the Bern Sex Role Inventory (BSRI) may be attributed to differences in the instructions and anchored rating scales used in the two studies. The present study tested the hypothesis that presence of various interaction effects involving instructions and rating scales would influence the acceptability of items for the BSRI Masculinity and Femininity Scales. Results based on the evaluation of individual items by Bem's item selection criteria in each of the four experimental conditions obtained by systematically manipulating two instructions (Bem's and Edwards' instructions) and two rating scales (Bem's and Edwards' rating scales) and also those based on the analysis of variance of item mean desirability ratings from the four experimental conditions supported the hypothesis.

Measuring Traditional and Liberated Males' Attitudes

Two factor analytically derived scales which assess traditional and liberated male attitudes are reported. The scales of 29 items are based on the responses of 277 men. Criteria for item selection are presented with an analysis of variance of the effects of age, marital and parental status on scale responses.

The Effect of Guessing on Item Reliability under Answer-Until-Correct Scoring

The answer-until-correct (AUC) procedure re quires that examinees respond to a multiple-choice item until they answer it correctly. The examinee's score on the item is then based on the number of responses required for the item. It was expected that the additional responses obtained under the AUC procedure would improve reliability by pro viding additional information on those examinees who fail to choose the correct alternative on their first attempt. However, when compared to the zero- one (ZO) scoring procedure, the AUC procedure has failed to yield consistent improvements in relia bility. Using a modified version of Horst's model for examinee behavior, this paper compares the ef fect of guessing on item reliability for the AUC pro cedure and the ZO procedure. The analysis shows that the relative efficiency of the two procedures de pends strongly on the nature of the item alterna tives and implies that the appropriate criteria for item selection are different for each procedure. Conflicting results reported for empirical compari sons of the reliabilities of the two procedures may result from a failure to control for the characteris tics of the items.

Using Item Test-Retest Stability (ITRS) as a Criterion for Item Selection: an Empirical Study

This paper proposes another criterion for empirical item-selection, namely, item test-retest stability (ITRS). Four tests were used in this research. Two of them were ability power tests and the other two were objective personality scales. These tests were administered twice to 100 students with a time interval of eight months in between. For each item in each test a phi correlation between the first and second administrations was calculated and then used as an ITRS index. For each test an abbreviated version was created, by selecting the items with the highest ITRS scores. The retest stability coefficients of the abbreviated and original tests were assessed with a new sample, consisting of another 100 students, and the abbreviated tests were found to be not lower than the respective coefficients of the original, longer tests. The ITRS and the more popular IIC (Item internal consistency) criterion were compared to each other in terms of their effects upon retest stability and internal consistency, and conclusions were drawn.

A Replication Study of Item Selection for the Bem Sex Role Inventory

An attempt was made to replicate the selection of items for the Bem Sex Role Inventory (BSRI). The 20 masculine, 20 feminine, and 20 neutral items of the BSRI were rated for social desirability "in an American male" and "in an American female" by male and female judges. The BSRI item-selection criterion—each item being rated by both male and female judges as significantly more desirable in a male than in a female (masculine items) or signifi cantly more desirable in a female than in a male (feminine items)—was met by only two items: mas culine and feminine. For a considerable number of other items, differences between mean desirability ratings for a male and for a female were in a direc tion opposite to that predicted. Correlations be tween the mean ratings of male and female judges when rating items for the same sex were quite high, consistent with previous research.

A Research Note on the MMPI Social I.E. Scale

Drake (1) has developed a social introversion scale for the Minnesota Multiphasic Personality Inventory, using scores on the Minnesota T.S.E. in ventory (3) as a criterion for item selection. In a second paper considering the validity of the new Si scale (2), it was shown that the scale would dis tinguish female university students engaging in 4, 5, or 6 types of activities from those engaging in 0, 1, or 2 types, and those participating in none from those participating in two or more. Evidence was also presented in this article testifying to the legitimacy of using number of extra-curricular activities as an index of social extroversion (2, p 552). A recently derived MMPI scale for socio-economic status (4) has given a correlation of -.58 with Si (5), suggesting that students of higher socio-economic status tend to secure scores at the extraverted end of the Si continuum.