Response.

Abstract

Dear Editor-in-Chief: The purpose of our article (5) was to present a detailed examination of “magnitude-based inference” (MBI) as a statistical method. Scientific evaluation of any proposed statistical method requires a precise description of the method, demonstration of properties of the method by deriving explicit formulas and/or implementing structured computer simulations, and careful interpretation of these results. We carried out these steps and demonstrated that MBI is a flawed method. Batterham and Hopkins (2) accused us of practicing “mathematistry,” that is, changing the problem and using mathematics gratuitously (3). This accusation is based on selective reading of Little’s article (3), which uses and displays far more mathematics than our article (5). We used minimal mathematics to give an unambiguous description of MBI as implemented in the spreadsheets [not hitherto available; see (5) supplemental digital content] and to evaluate its properties, neither of which is possible without using some mathematics. We did not redefine or modify the calculations in any way but simply presented and evaluated MBI calculations. As Little (3) wrote, “mathematics is the indispensable foundation of statistics” and “we need clear thinking mathematical statisticians to bring theory and rigour to our subject.” In evaluating MBI as a statistical method, it should be held to the same standards of clarity and rigor as other established statistical methods. Batterham and Hopkins (2) claimed that MBI is “philosophically and statistically distinct” from hypothesis testing, implying that it cannot be evaluated as a test. Our article (5) examined the method that they actually constructed both as a frequentist method and as a Bayesian method. We showed that MBI does not replace the use of P values by direct inference about magnitudes as claimed but rather uses two different probabilities that are interpretable either as P values for two different nonstandard tests or as approximate Bayesian calculations. These produce a test in both frameworks because the outcome is one of a simple set of possible categories. Despite their intention to move away from hypothesis testing, Batterham and Hopkins constructed a test—a test with demonstrably poor properties (5). MBI is not an “ideal Bayesian–frequentist hybrid” as Batterham and Hopkins (2) claimed because hybrids should have good properties in both frameworks, and MBI has poor frequentist properties (5). Arguably, from Rubin’s article (4), this also makes MBI a poor “reference Bayesian method”; see also Barker and Schofield (1). Methods like MBI that have demonstrably poor properties actually detract from, rather than add to, statistical methodology. Batterham and Hopkins (2) claimed that they have verified, “using analytical formulas and simulation,” that MBI Type 1 error rates “are much less than Welsh and Knight presented.” This claim is based on their own different definition of finding an effect. We were explicit about our simulation setting and what we did. Batterham and Hopkins (2) need to be equally explicit, use standard definitions correctly, and state their formulas, simulation settings, etc. Similarly, they claimed that their sample size calculations are correct. We made explicit the formula used in their spreadsheet to compute sample size and carefully explained why it is meaningless and should not be used. They responded by carrying out the same calculation (i.e. substituting inappropriate values into the standard sample size formula) using different software. This does not validate their approach because the issue is their misuse of the standard formula, not the software. We demonstrated (5) that MBI is a flawed statistical method and recommended that it not be used in practice. Everything we wrote (5) is justified and withstands the criticism of Batterham and Hopkins (2). Alan H. Welsh Mathematical Sciences Institute Australian National University Canberra, AUSTRALIA Emma J. Knight Performance Research Australian Institute of Sport Belconnen, AUSTRALIA