Null Hypothesis Significance Testing Procedure Research Articles

A taxonomy of model assumptions on which P is based and implications for added benefit in the sciences

ABSTRACTAlthough the null hypothesis significance testing procedure is problematic, many still favor the use of p-values as indicating the state of evidence against the model used to generate the p-value. From this perspective, p-values benefit science; or would benefit science if used correctly. In contrast, the novel argument to be presented introduces a taxonomy of assumptions included in the model; such as theoretical assumptions, auxiliary assumptions, statistical assumptions, and inferential assumptions. Careful attention is paid to the different categories of assumptions that models necessarily include to render p-value calculations sensible. Such careful attention suggests unappreciated limitations of p-values. Considering these limitations in the context of the descriptive statistics researchers routinely have available to them clarifies that p-values provide no added benefit to the scientist, above and beyond such descriptive statistics. The lack of added value, combined with the obvious harms documented in recent reviews, suggests that researchers in the sciences should rarely, or never, use p-values. Not even for indicating the state of evidence against models.

Dowodzenie hipotez za pomocą czynnika bayesowskiego (bayes factor): przykłady użycia w badaniach empirycznych

Statistical tests are used in science in order to support research hypotheses (theory, model). The Bayes Factor (BF) is a method that weighs evidence and shows which out of two hypotheses is better supported. Adopting the BF in statistical inference, we can show whether data provided stronger support for the null hypothesis, the alternative hypothesis or whether it is inconclusive and more data needs to be collected to provide more decisive evidence. Such a symmetry in interpretation is an advantage of the Bayes Factor over classical null hypothesis signifi cance testing (NHST). Using NHST, a researcher draws conclusions indirectly, by rejecting or not rejecting the null hypothesis. The discrepancy between these decisions and the researcher’s needs, often leads to misinterpretation of signifi cance test results, e.g. by concluding that non-signifi cant p-values are evidence for the absence of differences between groups or that variables are independent. In this work we show the main differences between the Bayesian and the frequential approach to the understanding of probability and statistical inference. We demonstrate how to verify hypotheses using the BF in practice and provide concrete examples of how it modifi es conclusions about empirical fi ndings based on the NHST procedure and the interpretation of p-values. We discuss the advantages of the BF – particularly the validation of a null hypothesis. Additionally, we provide some guidelines on how to do Bayesian statistics using the freeware statistical program JASP 0.8.

Using the Coefficient of Confidence to Make the Philosophical Switch From A Posteriori to A Priori Inferential Statistics.

There has been much controversy over the null hypothesis significance testing procedure, with much of the criticism centered on the problem of inverse inference. Specifically, p gives the probability of the finding (or one more extreme) given the null hypothesis, whereas the null hypothesis significance testing procedure involves drawing a conclusion about the null hypothesis given the finding. Many critics have called for null hypothesis significance tests to be replaced with confidence intervals. However, confidence intervals also suffer from a version of the inverse inference problem. The only known solution to the inverse inference problem is to use the famous theorem by Bayes, but this involves commitments that many researchers are not willing to make. However, it is possible to ask a useful question for which inverse inference is not a problem and that leads to the computation of the coefficient of confidence. In turn, and much more important, using the coefficient of confidence implies the desirability of switching from the current emphasis on a posteriori inferential statistics to an emphasis on a priori inferential statistics.

Performing Inferential Statistics Prior to Data Collection.

Typically, in education and psychology research, the investigator collects data and subsequently performs descriptive and inferential statistics. For example, a researcher might compute group means and use the null hypothesis significance testing procedure to draw conclusions about the populations from which the groups were drawn. We propose an alternative inferential statistical procedure that is performed prior to data collection rather than afterwards. To use this procedure, the researcher specifies how close she or he desires the group means to be to their corresponding population means and how confident she or he wishes to be that this actually is so. We derive an equation that provides researchers with a way to determine the sample size needed to meet the specifications concerning closeness and confidence, regardless of the number of groups.

Psychology and Statistics, Continued …: Readers Respond to the BASP Ban on P-Values

Abstract Andrew Gelman is not alone in questioning the way p-values are used in psychology. Earlier this year the journal Basic and Applied Social Psychology (BASP) decided to ban the null hypothesis significance testing procedure (NHSTP) from the papers it published. The decision was motivated by “the logical invalidity of the NHSTP”, said BASP editor David Trafimow – his argument being that p-values say nothing about the probability of a null hypothesis being false, so they should not be used to reject it. Our April report of the story prompted a handful of responses from readers, two of which are published below. Readers respond to the BASP ban on p-values

Auxiliary Assumptions and Epistemic Ratios: Fun with Bayes

Although there have been many Bayesian analyses of the null hypothesis significance testing procedure, Bayesian thinking has not yet made strong inroads into theory testing (as opposed to hypothesis testing). We explain why this may be and how to introduce Bayesian thinking into theory testing. Bayesian evidence often comes in the form of empirical support for a hypothesis that was derived from a larger theory. We show that the probability of finding an empirical hypothesis to be true when a theory is true, and the probability of finding an empirical hypothesis to be true when a theory is false, are both fundamental in rigorous scientific knowledge accumulation. In turn, however, these conditional probabilities depend on the quality of the auxiliary assumptions that researchers use to traverse the distance from a more general theory to an empirical hypothesis. Therefore, we provide Bayesian analyses of theory testing that feature auxiliary assumptions. Our analyses implicate the importance of creativity o...

An Expected Value Analysis of When to Avoid Type 1 and Type 2 Statistical Errors in Applied Research

For decades, applied researchers have relied heavily on either the null hypothesis significance testing procedure (NHSTP) or Bayesian statistics to help them determine the probability of the null hypothesis given their findings. While there may be some flaws in both methods, there are cases where the probability of the null hypothesis can be known, or at least estimated fairly accurately, thus making such analyses worthwhile. Assuming the worth of using such analyses, another issue in play is whether or not to avoid Type 1 or Type 2 statistical errors. While much has been said about the importance of avoiding both types of errors, we offer a new way of looking at the situation. We use an Expected Value analysis to help applied researchers determine when to avoid either type of error. We give examples and perform simulations to back up our mathematical arguments. Theoretical and applied issues are discussed.

Descriptive vs. inferential cheating

Descriptive vs. inferential cheating

To disconfirm or not to disconfirm: a null prediction vs. no prediction

To disconfirm or not to disconfirm: a null prediction vs. no prediction

An alternative statistical procedure in evaluating applied research products

For decades, applied researchers have had a choice between statistical methods that have been shown to not be perfectly valid (null hypothesis significance testing procedure) or to be based on premises that are difficult to justify (Bayesian method). These limitations have slowed the progress of product development, particularly in safety research. We propose a new statistical method termed epistemic ratio procedure (ERP), which is both logically valid and based on accurate premises that can be generated by either consumers or designers. This procedure allows applied researchers to compare the degree of empirical support among competing products or services. Theoretical and methodological implications for future applied research are discussed.

Korn and Freidlin's Misunderstanding of the Null Hypothesis Significance Testing Procedure

Korn and Freidlin's Misunderstanding of the Null Hypothesis Significance Testing Procedure

How often is prep close to the true replication probability?

Largely due to dissatisfaction with the standard null hypothesis significance testing procedure, researchers have begun to consider alternatives. For example, Killeen (2005a) has argued that researchers should calculate prep that is purported to indicate the probability that, if the experiment in question were replicated, the obtained finding would be in the same direction as the original finding. However, Killeen also seems to indicate that rather than being the probability of replication, prep is actually the probability of obtaining a finding whereby the experimental group mean exceeds the control group mean. Our goal was to determine the relative frequency with which obtained prep statistics are close to true replication probabilities. Regardless of which way prep is defined, our simulations show that it is unlikely to be close to the true value unless both the population effect magnitude and the sample size are uncommonly large. The definitional problem in combination with the inaccuracy under either interpretation, constitutes an important challenge for those who espouse the routine computation of prep statistics.

The Quantitative-Qualitative Distinction and the Null Hypothesis Significance Testing Procedure

Conventional discussion of research methodology contrast two approaches, the quantitative and the qualitative, presented as collectively exhaustive. But if qualitative is taken as the understanding of lifeworlds, the two approaches between them cover only a tiny fraction of research methodologies; and the quantitative, taken as the routine application to controlled experiments of frequentist statistics by way of the Null Hypothesis Significance Testing Procedure, is seriously flawed. It is contrary to the advice both of Fisher and of Neyman and Pearson, the two sources from which it is drawn.

Using Epistemic Ratios to Evaluate Hypotheses: An Imprecision Penalty for Imprecise Hypotheses

According to Bayesians, the null hypothesis significance-testing procedure is not deductively valid because it involves the retention or rejection of the null hypothesis under conditions where the posterior probability of that hypothesis is not known. Other criticisms are that this procedure is pointless and encourages imprecise hypotheses. However, according to non-Bayesians, there is no way of assigning a prior probability to the null hypothesis, and so Bayesian statistics do not work either. Consequently, no procedure has been accepted by both groups as providing a compelling reason to accept or reject hypotheses. The author aims to provide such a method. In the process, the author distinguishes between probability and epistemic estimation and argues that, although both are important in a science that is not completely deterministic, epistemic estimation is most relevant for hypothesis testing. Based on this analysis, the author proposes that hypotheses be evaluated via epistemic ratios and explores the implications of this proposal. One implication is that it is possible to encourage precise theorizing by imposing a penalty for imprecise hypotheses.

Hypothesis testing and theory evaluation at the boundaries: surprising insights from Bayes's theorem.

Because the probability of obtaining an experimental finding given that the null hypothesis is true [p(F\H0)] is not the same as the probability that the null hypothesis is true given a finding [p(H0\F)], calculating the former probability does not justify conclusions about the latter one. As the standard null-hypothesis significance-testing procedure does just that, it is logically invalid (J. Cohen, 1994). Theoretically, Bayes's theorem yields p(H0\F), but in practice, researchers rarely know the correct values for 2 of the variables in the theorem. Nevertheless, by considering a wide range of possible values for the unknown variables, it is possible to calculate a range of theoretical values for p(H0\F) and to draw conclusions about both hypothesis testing and theory evaluation.

Costs and benefits of statistical significance tests

Chow's book provides a thorough analysis of the confusing array of issues surrounding conventional tests of statistical significance. This book should be required reading for behavioral and social scientists. Chow concludes that the null-hypothesis significance-testing procedure (NHSTP) plays a limited, but necessary, role in the experimental sciences. Another possibility is that – owing in part to its metaphorical underpinnings and convoluted logic – the NHSTP is declining in importance in those few sciences in which it ever played a role.