Quantitative subjective assessments are almost always biased, sometimes completely misleading*

Abstract

Subjective assessments are used widely in civilized societies for evaluating people, the work they do, and the stresses they work under. Yet most of the people who use subjective assessments are probably not aware of the pitfalls which they involve.Range effects introduced by the investigator almost inevitably bias quantitative subjective assessments, unless very special precautions are taken to prevent or eliminate the bias. Stevens' method of direct magnitude estimation is always biased by range effects. The international standard for calculating the loudness of noise corresponds to a 30 decibel range of sound intensity.When the investigator takes special precautions not to introduce bias, the observers may use their own well‐known standards which they bring with them to the investigation. In a height constancy investigation conducted in a field, the well‐known standard is the height of a hedge or fence. In judging just acceptable noise levels, the well‐known standards used by the observers must depend upon the noise levels which they are accustomed to.The chief danger of subjective assessments is that they may be based upon a well‐known rule which does not happen to apply in the particular circumstance of the investigation. A result may then be obtained which is the direct opposite of the truth. Examples are the well‐known rules that noise and heat interfere with work. Yet performance can improve in both noise and heat. Results of this kind are revealed by measuring performance, not by asking people how noise or heat affects them. Subjective assessments should be used to complement measures of performance, not to replace them.