Impact of Unreliability of Measurements on Statistical Conclusion Validity

Abstract

The validity of research study results is always of concern because validity addresses the issue of the truth or falsity of the hypotheses tested. At the conclusion of a study after the statistical analyses have been completed, it is desirable to have a high level of confidence that the results approximate the truth about the relationship between the independent and dependent variables studied. The reliability (or unreliability) of measurements of the independent and dependent variables has a direct impact on the validity of study results because unreliable measures decrease statistical conclusion validity. When statistical conclusion validity is compromised, one cannot make highly valid inferences based on the results of statistical tests. Statistical conclusion validity is one of the four major types of study validity that need to be addressed within a research study. (The other types of validity include internal validity, external validity, and construct validity of putative causes and effects.) Campbell and Stanley (1963) and Cook and Campbell (1979) have published classic works that discuss the various types of validity that need to be carefully considered within the context of a research study, and I highly recommend a careful reading of them. There are three measurement-related threats to statistical conclusion validity. These are: 1. Unreliable measures, 2. Unreliable treatment implementation within an experimental or quasi-experimental design, and 3. Random irrelevancies in the experimental setting. Let us consider the role of unreliable measures as a threat to statistical conclusion validity. Tests of statistical hypotheses depend upon covariation between independent and dependent variables for inferring relationships or cause, and test the question: "Are the presumed independent and dependent variables related?" (Cook & Campbell, 1979, p. 37). When measurement of any of the variables in a hypothesized relationship is unreliable, then false conclusions about covariation of variables can be made based on statistical evidence. In other words, unreliable measures introduce more random error into scores and into the tested relationship between variables. Random error results in data "instability." Unstable sample data on any of the variables tested in a statistical hypothesis can result in spuriously high or low sample means or variation between group scores, which can lead to statistical results that can lead to false conclusions about population covariation. In essence, conclusions can be drawn based on the results of statistical analysis that are not true because measures with low reliability are unlikely to register true scores. Of course the main way to control for unreliability of measures is to carefully select measures that have prior evidence for their reliability in the population that will be used for the study. In addition, reliability should be assessed using data collected for the present study whenever possible to get an assessment of the approximate amount of random error variance. Internal consistency analysis can be conducted on questionnaires with scaled items or Likert-type items, and test-retest analysis can be done on a subsample of subjects within the study for variables that should remain stable over time. When the independent variable in a study is a treatment intervention, there is the potential for a threat to statistical conclusion validity due to a poorly standardized and controlled intervention protocol. Reduced standardization of an intervention is more likely to result when more than one person is delivering the intervention to subjects because different interveners are likely to deliver the intervention differently. …