Automated Content Analysis of Multiple-Choice Test Item Banks

Abstract

Test item review is a specialized type of content analysis conducted to identify and correct test item flaws early in the test development process. Test item reviewers not only examine the targeted content of a test but also remove inappropriate content and balance various types of incidental content. An automated content analysis implementation of Hiller’s verbal ambiguity scales and Laffal’s General Concept Dictionary of English was used to examine 576 multiple-choice test items before and after test item review and revision by experienced item editors. Hiller’s scales detected some problems with item clarity. Laffal’s categories detected content imbalance between test forms but not inappropriate item content.