Reliability of Tests: A Technique Using the “Regression to the Mean” Fallacy

Abstract

There are many situations in empirical research where an receives a on a particular test, e.g., students taking IQ or aptitude tests, consumers giving a rating to a particular product, golfers playing 18 holes of golf on a particular course. In all these cases the test score will consist of a measure of true ability plus an error component. In this article we present a method of separating the variance of the scores into a component that is due to the true difference of ability and a component that is due to the error (or unreliability) of the test. Obviously, if each individual takes the test only once there is nothing we can do to separate the ability variance and test variance. A histogram of the test scores will have a variance, but it will be a combination of the two components. If each person could take the test many times, there would be no problem; an analysis of variance could be performed. The within-person variance would measure the reliability of the test and the between-person variance would measure the variance of true ability in the population. However, this type of analysis is not always possible. The individual may learn as he takes the same test over and over again, subjects may become bored, time may not permit many re-tests, etc. This article presents a methodology in which each person takes the same test twice. Our technique will also give some insight into the familiar to the mean fallacy and allows us to quantify this regression fallacy. In getting a person to take the same test (e.g., answer the same question) twice, there may be a number of methodological problems. He may remember his answer on the first trial and wish to appear consistent; the context of the question may have changed; in the intervening time period all other things may not have remained equal. Therefore, the model that we develop in this article has a noise or unreliability component that could contain some of the above factors. Hence, care should be taken to assure that the second trial is as similar to the first trial as possible so that the noise component of the model gives an accurate indication of the reliability of the test. The model only assesses the reliability of the measuring technique-the validity of the scores is not discussed at all. In fact, the model that will be developed has no implactions at all for validity. Some obvious marketing applications of the model are in the areas of consumers' ratings of products or product attributes, subjects' statements concerning perceived similarities of brands, and virtually any other area where a subject gives a score to an object with respect to an attribute.