Abstract
This paper investigates statistical reproducibility of the -test. We formulate reproducibility as a predictive inference problem and apply the nonparametric predictive inference method. Within our research framework, statistical reproducibility provides inference on the probability that the same test outcome would be reached, if the test were repeated under identical conditions. We present an nonparametric predictive inference algorithm to calculate the reproducibility of the -test and then use simulations to explore the reproducibility both under the null and alternative hypotheses. We then apply nonparametric predictive inference reproducibility to a real-life scenario of a preclinical experiment, which involves multiple pairwise comparisons of test groups, where different groups are given a different concentration of a drug. The aim of the experiment is to decide the concentration of the drug which is most effective. In both simulations and the application scenario, we study the relationship between reproducibility and two test statistics, the Cohen’s and the -value. We also compare the reproducibility of the -test with the reproducibility of the Wilcoxon Mann–Whitney test. Finally, we examine reproducibility for the final decision of choosing a particular dose in the multiple pairwise comparisons scenario. This paper presents advances on the topic of test reproducibility with relevance for tests used in pharmaceutical research.
Highlights
Statistical Methods in Medical Research XX(X)Reproducibility of tests is a complex issue, which is of importance in pharmaceutical research and development
The nonparametric predictive inference (NPI) reproducibility probability is the probability for the event that, if a test were repeated under identical circumstances and with the same sample size, the same test outcome would be reached
5 Reproducibility of the final decision based on multiple pairwise comparisons In Section 3 we introduced NPI bootstrap (NPI-B)-RP for the t-test for the comparison of two groups and in Section 4 we presented NPI-B-RP for pairwise comparisons in a pharmaceutical test scenario
Summary
Reproducibility of tests is a complex issue, which is of importance in pharmaceutical research and development. Goodman used the term replication probability rather than reproducibility probability, his definition is very similar to the definition of reproducibility adopted in this paper He defined it as the probability of observing another statistically significant result in the same direction as the first one, if an experiment was repeated under identical conditions and with the same sample size. This final decision is of interest as in practice decisions are often based on more than one single statistical test; studying its reproducibility is important and to date has received little attention in the literature. All calculations have been done using R version 3.2.4, the code is available from the link https://tahanimaturi.com/rcodes/Rcodes-SMMR-May-2021.zip
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.