Obtaining Test Blueprint Weights From Job Analysis Surveys

Abstract

A method for combining multiple scale responses from job or task surveys based on a hierarchical ranking scheme is presented. A rationale for placing the resulting ordinal information onto an interval scale of measurement using the Rasch Rating Scale Model is also provided. After a simple linear transformation, the item or task parameter estimates can be used to obtain item weights to be used in constructing test blueprints. Prior weights can then be used to modify the item weights after data collection, based either on content balancing requirements or Bayesian prior content weights from SMEs (subject matter experts). Finally a method is suggested to link two or more surveys, again using the Rasch Rating Scale Model and the computer program, Bigsteps, when it is desirable to shorten the length of the typical job or task survey.