Abstract
Several reproducibility probability (RP)-estimators for the binomial, sign, Wilcoxon signed rank and Kendall tests are studied. Their behavior in terms of MSE is investigated, as well as their performances for RP-testing. Two classes of estimators are considered: the semi-parametric one, where RP-estimators are derived from the expression of the exact or approximated power function, and the non-parametric one, whose RP-estimators are obtained on the basis of the nonparametric plug-in principle. In order to evaluate the precision of RP-estimators for each test, the MSE is computed, and the best overall estimator turns out to belong to the semi-parametric class. Then, in order to evaluate the RP-testing performances provided by RP estimators for each test, the disagreement between the RP-testing decision rule, i.e., “accept H0 if the RP-estimate is lower than, or equal to, 1/2, and reject H0 otherwise”, and the classical one (based on the critical value or on the p-value) is obtained. It is shown that the RP-based testing decision for some semi-parametric RP estimators exactly replicates the classical one. In many situations, the RP-estimator replicating the classical decision rule also provides the best MSE.
Highlights
Statistical tests are usually applied in almost all fields of science to evaluate experimental results
The reproducibility probability (RP) is the true power of a statistical test, and its estimation provides useful information to evaluate the stability of statistical test results
RP-testing, that is the adoption of the RP estimate to evaluate the significance of statistical test results, can substitute the p-value testing [7,8]
Summary
Statistical tests are usually applied in almost all fields of science to evaluate experimental results. We argue that the RP-testing rule can be adopted in order to bypass the many, well-known criticisms raised by the p-value [9,10,11,12,13] In the context of nonparametric tests, RP estimation has not yet been widely studied. The sign test, the binomial test, the Kendall test and the Wilcoxon signed rank test are considered RP estimation and testing for the Wilcoxon signed rank test is studied, where semi-parametric and nonparametric plug-in estimators are considered and studied separately; the behavior of different estimators is compared through simulation. As in the previous sections, semi-parametric and nonparametric estimators are studied separately, a simulation is run to compare the behavior of different estimators, in terms of MSE and RP-testing performances.
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
