Latent Trait Estimates Research Articles

Computerized Adaptive Testing of Personality Traits

A computerized adaptive testing (CAT) procedure was simulated with ordinal polytomous personality data collected using a conventional paper-and-pencil testing format. An adapted Dutch version of the dominance scale of Gough and Heilbrun’s Adjective Check List (ACL) was used. This version contained Likert response scales with five categories. Item parameters were estimated using Samejima’s graded response model from the responses of 1,925 subjects. The CAT procedure was simulated using the responses of 1,517 other subjects. The value of the required standard error in the stopping rule of the CAT was manipulated. The relationship between CAT latent trait estimates and estimates based on all dominance items was studied. Additionally, the pattern of relationships between the CAT latent trait estimates and the other ACL scales was compared to that between latent trait estimates based on the entire item pool and the other ACL scales. The CAT procedure resulted in latent trait estimates qualitatively equivalent to latent trait estimates based on all items, while a substantial reduction of the number of used items could be realized (at the stopping rule of 0.4 about 33% of the 36 items was used).

A Randomized Experiment to Compare Conventional, Computerized, and Computerized Adaptive Administration of Ordinal Polytomous Attitude Items

A total of 520 high school students were randomly assigned to a paper-and-pencil test (PPT), a computerized standard test (CST), or a computerized adaptive test (CAT) version of the Dutch School Attitude Questionnaire (SAQ), consisting of ordinal polytomous items. The CST administered items in the same order as the PPT. The CAT administered all items of three SAQ subscales in adaptive order using Samejima’s graded response model, so that six different stopping rule settings could be applied afterwards. School marks were used as external criteria. Results showed significant but small multivariate administration mode effects on conventional raw scores and small to medium effects on maximum likelihood latent trait estimates. When the precision of CAT latent trait estimates decreased, correlations with grade point average in general decreased. However, the magnitude of the decrease was not very large as compared to the PPT, the CST, and the CAT without the stopping rule.

Implementation and Measurement Efficiency of Multidimensional Computerized Adaptive Testing

Multidimensional adaptive testing (MAT) procedures are proposed for the measurement of several latent traits by a single examination. Bayesian latent trait estimation and adaptive item selection are derived. Simulations were conducted to compare the measurement efficiency of MAT with those of unidimensional adaptive testing and random administration. The results showed that the higher the correlation between latent traits, the more latent traits there were, and the more scoring levels there were in the items, the more efficient MAT was than the other two procedures. For tests containing multidimensional items, only MAT is applicable, whereas unidimensional adaptive testing is not. Issues in implementing MAT are discussed.

An EM Approach to Parameter Estimation for the Zinnes and Griggs Paired Comparison IRT Model

Borman et al. recently proposed a computer adaptive performance appraisal system called CARS II that utilizes paired comparison judgments of behavioral stimuli. To implement this approach,the paired comparison ideal point model developed by Zinnes and Griggs was selected. In this article,the authors describe item response and information functions for the Zinnes and Griggs model and present procedures for estimating stimulus and person parameters. Monte carlo simulations were conducted to assess the accuracy of the parameter estimation procedures. The results indicated that at least 400 ratees (i.e.,ratings) are required to obtain reasonably accurate estimates of the stimulus parameters and their standard errors. In addition,latent trait estimation improves as test length increases. The implications of these results for test construction are also discussed.

Item response theory and health outcomes measurement in the 21st century.

Item response theory (IRT) has a number of potential advantages over classical test theory in assessing self-reported health outcomes. IRT models yield invariant item and latent trait estimates (within a linear transformation), standard errors conditional on trait level, and trait estimates anchored to item content. IRT also facilitates evaluation of differential item functioning, inclusion of items with different response formats in the same scale, and assessment of person fit and is ideally suited for implementing computer adaptive testing. Finally, IRT methods can be helpful in developing better health outcome measures and in assessing change over time. These issues are reviewed, along with a discussion of some of the methodological and practical challenges in applying IRT methods.

Latent trait estimates of rater reliability in the international English Language Testing System (IELTS)

Abstract The study examined the effects of fixed criteria, training and moderation on reliability of ratings assigned to written scripts. Using an item response analysis, consistency of inter and intra rater reliability of scoring patterns were examined under changing conditions. Ratings were assigned twice under workshop conditions and once under unsupervised isolated conditions. The workshops were used to identify criteria used by raters and then to obtain an agreed set of criteria using a consensus moderation approach. Results indicate that raters are influenced by their backgrounds, the moderation procedure and by the criteria depending on the circumstances under which the ratings were assigned. However a lack of fit of the ratings to a single dimension model over time, suggests that the raters may change their criteria under different conditions. Although similar ratings may be assigned, different criteria are employed by the same rater over time. The results seriously question the use of classical measurement approaches in the assessment of rater reliability.

Adaptive Testing: a Bayesian Procedure for the Efficient Measurement of Ability

Abstract: The use of response‐contingent testing to estimate mental abilities has relied hitherto on pre‐programming of items. An obvious extension is to perform the item selection during the test administration, making use of information about the subject as he provides it The updating of ability estimates is handled naturally by repeated application of Bayes theorem. Using a probablistic item response model, a latent trait estimation procedure is developed. Results of an actual implementation of adaptive testing are reported and applications of the procedure in a number of measurement settings are suggested.