Computer-assisted versus paper-and-pencil administration of questionnaires

Abstract

In a field experiment using 95 employees, the results of computer-assisted versus paper-and-pencil administration of questionnaires during computer training were compared. Unlike past research, the computer mode of administration was made as similar as possible, in format, to the paper mode. No differences in means or reliabilities were found between any variables across the two conditions. However, when the correlations were examined, different patterns of relationships were found across the two conditions. Specifically, subjective or affective measures (software efficacy, playfulness, and training reaction) were more highly related when the measures were gathered by computer as op­ posed to paper-and-pencil. The relationships between subjective measures and more objective or factual measures (experience, performance, training transfer) were not affected. Thus, it appears that, while differences in mode of administration may not be reflected in differences in means, there are differences in nomological validity. Organizations of various kinds have traditionally col­ lected questionnaire data through paper-and-pencil ad­ ministration, and most validation studies of these ques­ tionnaire instruments have been conducted using paper and pencil. However, computer-assisted testing of job applicants and employees is becoming increasingly pop­ ular in organizations and in research (Booth-Kewley, Ed­ wards, & Rosenfeld, 1992). Computer-assisted testing has several advantages over paper-based testing: reduced costs and increased availability of tests (Newsted, 1988), au­ tomatic tabulation and analysis ofdata (Martin & Nagao, 1989), higher test security and lower ambiguity of re­ sponses (Federico, 1992), easier generation ofcustomized questionnaires (Greenwood, Higgins, & Dimnik, 1987) and assessment ofcertain individual abilities that cannot be measured through more traditional paper-and-pencil tests (Drasgow, Olson, Keenan, Moberg, & Mead, 1993; Jones, Dunlap, & Bilodeau, 1986; Schmitt, Gilliland, Lan­ dis, & Devine, 1993). Translation ofpaper-based questionnaires to computer­ based formats represents a significant change in mea­ surement, and the equivalence of the measure collected by this new method must be assessed. Research compar