RUM leads to noise: the significance of finding the sources of variability between experimental runs

Abstract

Scientists typically run experiments many times to find general patterns over multiple specific runs. The results of those runs vary, and the variance is often simply referred to as “noise”. We claim that it is highly important to separate the components that contribute to noise and to recognize to which degree they contribute to it. Consideration of the relative contributions of R (randomness), U (uncontrolled variables), and M (measurement error) helps to interpret data and can help to improve experimental designs. We explain this using a hypothetical example and point out that assumptions of what causes variability in the results of experiments are often made implicitly. Further, we demonstrate our point by showing how it can change the interpretation of real data. Because of a lack of explicit discussion of underlying assumptions, it is possible that sources of noise are misidentified to be either existent or non-existent. This can happen if, for example, measurement error is assumed when there is none, an assumption that would mask a real effect that could deserve further study. Despite these factors, the contribution of different factors to the overall noise is rarely considered, hampering scientific progress.