Regarding sources of irreducible variation in transformed up-down staircase experiments

Abstract

A modicum of variation in the result is always expected when a psychoacoustic experiment is run multiple times on a single subject. Historically, this variation was attributed to human shortcomings: lapses in attention, inexperience, and the effects of memory, learning, and fatigue. It is demonstrated that the use of any psychophysical method that requires a binary response (i.e., a correct or incorrect response) will generate irreducible variation in the result—even for a perfect subject. At NASA, a recent battery of low-frequency simultaneous masking experiments was undertaken (see NASA-TM-2018-220120). This effort was preceded by a lengthy test design phase—also documented therein—during which it was determined that the test should consist of a large number of subjects, each to be tested minimally. For this, transformed up-down staircases were simulated in order to produce a prediction of the irreducible variance as a function of the staircases’ parameters (e.g., length in the number of trials). The results from repeated test conditions taken during the experiment indicate that at least half of the variation observed within subjects may be coming from this irreducible source, though the propensity to produce inconsistent results may be a feature of some subjects more than others.