The effect of scale manipulations on validity: Targetting frequency rating scales for anticipated performance levels

Abstract

A common problem with standard five-point frequency rating scales is their inability to differentiate between objects within a relatively narrow band of the rating dimension. Two alternatives for increasing a scale's ability to reflect existing differences are: increasing the number of positions on the rating scale, or packing the rating scale with quantifiers from a particular portion of the frequency dimension. In this study, three types of rating scale — a standard five-point balanced scale, a longer nine-point balanced scale, and a five-point packed scale — were used to rate two videotaped samples of behaviour, one displaying performance levels from 10% to 100% and the other from 70% to 90%. Each subject's ratings were correlated with true performance levels in each sample of behaviour as a measure of validity. Results showed that for ratings of the wide range of behaviour all three types of scales provided average correlations between the ratings and actual frequencies of the event which exceeded 0·90, with the longer nine-point scale yielding a significantly higher mean correlation than the other two scales. For ratings of the narrow performance range, the nine-point scale provided the highest correlation with the actual frequencies, followed by the packed scale and the standard five-point balanced scale. All differences were significant. Findings suggest that increasing the number of scale positions can significantly increase the validity of ratings obtained. Also, though to a lesser degree, validity of ratings may be enhanced in a shorter scale by using quantifiers from the portion of the frequency continuum where performance are anticipated to lie.