Improving reproducibility probability estimation and preserving RP-testing

Abstract

Experimental results are often interpreted through statistical tests, where the alternative hypothesis represents the theory to be evinced; if the experimental results lead to the rejection of the null hypothesis, the theory is supported by empirical evidence. In these cases, the reproducibility of this empirical evidence can be measured by the Reproducibility Probability (RP) of the test, which coincides with the probability of rejecting the null hypothesis. The terminology “Reproducibility” Probability stems from the fact that it is usually computed when an experiment provides a significant result to evaluate the probability that a further identical and independent experiment confirms the statistical significance. In recent literature, some RP estimators have been proposed. They are useful for two reasons: they allow us to evaluate the reliability of the obtained statistical significance and some estimates can be used as a test statistic, owing to the so-called “RP-testing” decision rule (reject the null hypothesis if and only if the RP estimate is greater than 1/2). Unfortunately, the usually adopted RP estimators are affected by a high mean squared error. In this paper, a new class of RP estimators is introduced and examined to improve their estimation precision. Specifically, the performances of the new RP estimators have been compared with those of the existing estimators and a 30% greater reduction in the mean squared error (on average) was observed. Moreover, the new estimator with the best performance allowed the use of the RP-testing decision rule. Hence, this work achieves the double goal of improving Reproducibility Probability estimation and preserving RP-testing.