The Influence of Item Modality on the Dimension Measured by a Test

Abstract

WHEN A battery of proficiency tests for bombing-navigation systems mechanics was val idated, one of the tests, designed to measure ability to read dials and indicators, failed to dis criminate significantly among mechanics of dif ferent levels of proficiency. A task-o rie nted analysis and the unanimous opinion of experts in systems maintenance both indicated that the abil ity to read dials and indicators is an important part of routine maintenance skill. In seeking to account for the apparent failure of the te s t to measure this ability, it was suggested that the form of the items might be the cause. The 34 items on the test are all in five alter native multiple choice form. Each item pre sents a drawing of one of the dials in the system with the hand pointing to a typical value. Four of the responses consist of more or less plaus ible readings, and the fifth a none of the above is correct' ' type of statement. The mechanic being tested is supposed to read the dial and in dicate which response agrees best with his read ing. Actually, what may be happening is that the mechanic is making comparisons between the responses and the picture of the dial, and elim inating some or all of the responses as obvious ly wrong. The task offered by the test probably does not resemble the task he must perform on the job, when, rather than choose among four readings, he must get the reading from the dial without aid of any sort. This possibility seemed strong enough to warrant investigation, so a second form of the test was constructed, consisting of the same 34 items in completion form?that is, with blanks in which the mechanic was to write the value on the dial as he read it. His reading was consid ered correct if it fell within accepted tolerance limits (for that dial) of the actual reading. A cross-validation study of the battery was contemplated in the future. In the meantime, a decision was made to administer the two forms to graduates of a training course for mechanics and to analyze their scores to see whether or not the completion form measures the same thing that the multiple-choice form measures. If it does measure the same thing, it is no better than the multiple-choice test and there will be no need to include either form in the battery to be cross validated; if it measures something else, it will be worthwhile to include it.