Null Hypothesis Significance Testing Research Articles

From significance testing to estimation and Open Science: How esci can help.

We argue that researchers should test less, estimate more, and adopt Open Science practices. We outline some of the flaws of null hypothesis significance testing and take three approaches to demonstrating the unreliability of the p value. We explain some advantages of estimation and meta-analysis ("the new statistics"), especially as contributions to Open Science practices, which aim to increase the openness, integrity, and replicability of research. We then describe esci (estimation statistics with confidence intervals): a set of online simulations and an R package for estimation that integrates into jamovi and JASP. This software provides (a) online activities to sharpen understanding of statistical concepts (e.g., "The Dance of the Means"); (b) effects sizes and confidence intervals for a range of study designs, largely by using techniques recently developed by Bonett; (c) publication-ready visualisations that make uncertainty salient; and (d) the option to conduct strong, fair hypothesis evaluation through specification of an interval null. Although developed specifically to support undergraduate learning through the 2nd edition of our textbook, esci should prove a valuable tool for graduate students and researchers interested in adopting the estimation approach. Further information is at ( https://thenewstatistics.com).

Classical tests, linear models and their extensions for the analysis of 2 × 2 contingency tables

Abstract Ecologists and evolutionary biologists are regularly tasked with the comparison of binary data across groups. There is, however, some discussion in the biostatistics literature about the best methodology for the analysis of data comprising binary explanatory and response variables forming a 2 × 2 contingency table. We assess several methodologies for the analysis of 2 × 2 contingency tables using a simulation scheme of different sample sizes with outcomes evenly or unevenly distributed between groups. Specifically, we assess the commonly recommended logistic (generalised linear model [GLM]) regression analysis, the classical Pearson chi‐squared test and four conventional alternatives (Yates' correction, Fisher's exact, exact unconditional and mid‐p), as well as the widely discouraged linear model (LM) regression. We found that both LM and GLM analyses provided unbiased estimates of the difference in proportions between groups. LM and GLM analyses also provided accurate standard errors and confidence intervals when the experimental design was balanced. When the experimental design was unbalanced, sample size was small, and one of the two groups had a probability close to 1 or 0, LM analysis could substantially over‐ or under‐represent statistical uncertainty. For null hypothesis significance testing, the performance of the chi‐squared test and LM analysis were almost identical. Across all scenarios, both had high power to detect non‐null effects and reject false positives. By contrast, the GLM analysis was underpowered when using z‐based p‐values, in particular when one of the two groups had a probability near 1 or 0. The GLM using the LRT had better power to detect non‐null results. Our simulation results suggest that, wherever a chi‐squared test would be recommended, a linear regression is a suitable alternative for the analysis of 2 × 2 contingency table data. When researchers opt for more sophisticated procedures, we provide R functions to calculate the standard error of a difference between two probabilities from a Bernoulli GLM output using the delta method. We also explore approaches to compliment GLM analysis of 2 × 2 contingency tables with credible intervals on the probability scale. These additional operations should support researchers to make valid assessments of both statistical and practical significances.

Equivalence of telehealth and face-to-face administration of the Wechsler Adult Intelligence Scale Fourth Edition (WAIS-IV)

Objective: Digital provision of cognitive tests like the Wechsler Adult Intelligence Scale Fourth Edition (WAIS-IV) has the potential to significantly increase access to important assessments. However, limited empirical evidence exists for the equivalence of telehealth and face-to-face administration. Presently test publishers recommend not administering subtests with stimulus materials that require manipulation via telehealth. Therefore, this study evaluated the equivalence of a telehealth administration procedure of the WAIS-IV with face-to-face administration. Method: A randomized repeated measures design included a sample of N = 28 participants with typical cognitive functioning, predominantly female (61%), aged 21 years (SD = 3.65), and identified as Australian (79%). Results: Our analysis showed that the point estimates of mean differences for indices and subtests (except PSI and Symbol Search) between face-to-face and telehealth applications were smaller than the smallest effect size of interest (SESOI). Analysis of 90% CIs around the mean difference showed the PRI Index and subtests Vocabulary, Information, and Arithmetic were statistically equivalent, while FSIQ, VCI, FRI, WMI indices, and other seven subtests were not statistically equivalent. For Null Hypothesis Significance Tests, the indices and all subtests were not significantly different from zero. Conclusions: These findings show a telehealth administration of the WAIS-IV provides scores similar to those collected in face-to-face administration, and observed differences were smaller than the difference expected due to measurement error. However, psychologists are cautioned not to solely rely on test scores when formulating outcomes but use clinical judgement with awareness of potential (albeit small) errors introduced by telehealth testing.

"Catastrophic" set size limits on infants' capacity to represent objects: A systematic review and Bayesian meta-analysis.

Decades of research has revealed that humans can concurrently represent small quantities of three-dimensional objects as those objects move through space or into occlusion. For infants (but not older children or adults), this ability apparently comes with a significant limitation: when the number of occluded objects exceeds three, infants experience what has been characterized as a "catastrophic" set size limit, failing to represent even the approximate quantity of the hidden array. Infants' apparent catastrophic representational failures suggest a significant information processing limitation in the first years of life, and the evidence has been used as support for prominent theories of the development of object and numerical cognition. However, the evidence for catastrophic failure consists of individual small-n experiments that use null hypothesis significance testing to obtain null results (i.e., p>0.05). Whether catastrophic representational failures are robust or reliable across studies, methods, and labs is not known. Here we report a systematic review and Bayesian meta-analysis to examine the strength of the evidence in favor of catastrophic representational failures in infancy. Our analysis of 22 experiments across 12 reports, with a combined total of n=367 infants aged 10-20 months, revealed strong support for the evidence for catastrophic set size limits. A complementary analysis found moderate support for infants' success when representing fewer than four objects. We discuss the implications of our findings for theories of object and numerical cognitive development. RESEARCH HIGHLIGHTS: Previous work has suggested that infants are unable to concurrently represent four or more objects-a "catastrophic" set size limit. We reviewed this work and conducted a Bayesian meta-analysis to examine the robustness of this limit across individual small-n experiments. We found strong support for the evidence for catastrophic set size limits, and moderate support for infants' success when representing fewer than four objects.

P-Values as QWERTY: Curating Evidence in the Computational Era

McShane et al.'s (2024) wide-ranging critique of null hypothesis significance testing provides a number of specific suggestions for improved practice in empirical research. This commentary amplifies several of these from the perspective of computational statistics—particularly nonparametrics, resampling/bootstrapping, and Bayesian methods—applied to common research problems. Throughout, the author emphasizes estimation (as opposed to testing) and uncertainty quantification through a comprehensive process of “curating” a variety of graphical and tabular evidence. Specifically, researchers should be encouraged to estimate the quantities that matter, with as few assumptions as possible, in multiple ways, then try to visualize it all, documenting their pathway from data to results for others to follow.

Introduction to Bayesian Analyses for Clinical Research.

Bayesian analyses are becoming more popular as a means of analyzing data, yet the Bayesian approach is novel to many members of the broad clinical audience. While Bayesian analyses are foundational to anesthesia pharmacokinetic/pharmacodynamic modeling, they also can be used for analyzing data from clinical trials or observational studies. The traditional null hypothesis significance testing (frequentist) approach uses only the data collected from the current study to make inferences. On the other hand, the Bayesian approach quantifies the external information or expert knowledge and combines the external information with the study data, then makes inference from this combined information. We introduce to the clinical and translational science researcher what it means to do Bayesian statistics, why a researcher would choose to perform their analyses using the Bayesian approach, when it would be advantageous to use a Bayesian instead of a frequentist approach, and how Bayesian analyses and interpretations differ from the more traditional frequentist methods. Throughout this paper, we use various pain- and anesthesia-related examples to highlight the ideas and statistical concepts that should be relatable to other areas of research as well.

Most Psychological Researchers Assume Their Samples Are Ergodic: Evidence From a Year of Articles in Three Major Journals

Conventional statistics methods in most psychological research, such as null-hypothesis significance tests (NHSTs), use aggregated values (i.e., the sample means) of group behaviours to make inferences about individuals. Such inferences are possibly erroneous because groups of humans rarely, if ever, constitute an ergodic system. To assume ergodicity without checking is to commit the ‘ergodic fallacy’. The aim of the current study was to examine the prevalence of this error in contemporary psychological research. We analysed three highly cited ‘Q1’ journals in the fields of clinical, educational and cognitive psychology for statements that indicated this error. As hypothesized, the ergodic fallacy was found in the vast majority of the papers investigated here. We also hypothesised that the prevalence of this error would be highest in cognitive psychology papers because this field typically assesses theoretical claims about universal cognitive mechanisms, whereas clinical and educational psychology are more concerned with empirically supported interventions. This hypothesis was also supported by our results. Nonetheless, the prevalence of the ergodic fallacy was still high in all fields. Implications are discussed with respect to the reporting of research findings and the validity of theories in psychology.

“Statistical Significance” and Statistical Reporting: Moving Beyond Binary

Null hypothesis significance testing (NHST) is the default approach to statistical analysis and reporting in marketing and the biomedical and social sciences more broadly. Despite its default role, NHST has long been criticized by both statisticians and applied researchers, including those within marketing. Therefore, the authors propose a major transition in statistical analysis and reporting. Specifically, they propose moving beyond binary: abandoning NHST as the default approach to statistical analysis and reporting. To facilitate this, they briefly review some of the principal problems associated with NHST. They next discuss some principles that they believe should underlie statistical analysis and reporting. They then use these principles to motivate some guidelines for statistical analysis and reporting. They next provide some examples that illustrate statistical analysis and reporting that adheres to their principles and guidelines. They conclude with a brief discussion.

Frequentist, Bayesian Analysis and Complementary Statistical Tools for Geriatric and Rehabilitation Fields: Are Traditional Null-Hypothesis Significance Testing Methods Sufficient?

Null hypothesis significant testing (NHST) is the dominant statistical approach in the geriatric and rehabilitation fields. However, NHST is routinely misunderstood or misused. In this case, the findings from clinical trials would be taken as evidence of no effect, when in fact, a clinically relevant question may have a "non-significant" p-value. Conversely, findings are considered clinically relevant when significant differences are observed between groups. To assume that p-value is not an exclusive indicator of an association or the existence of an effect, researchers should be encouraged to report other statistical analysis approaches as Bayesian analysis and complementary statistical tools alongside the p-value (eg, effect size, confidence intervals, minimal clinically important difference, and magnitude-based inference) to improve interpretation of the findings of clinical trials by presenting a more efficient and comprehensive analysis. However, the focus on Bayesian analysis and secondary statistical analyses does not mean that NHST is less important. Only that, to observe a real intervention effect, researchers should use a combination of secondary statistical analyses in conjunction with NHST or Bayesian statistical analysis to reveal what p-values cannot show in the geriatric and rehabilitation studies (eg, the clinical importance of 1kg increase in handgrip strength in the intervention group of long-lived older adults compared to a control group). This paper provides potential insights for improving the interpretation of scientific data in rehabilitation and geriatric fields by utilizing Bayesian and secondary statistical analyses to better scrutinize the results of clinical trials where a p-value alone may not be appropriate to determine the efficacy of an intervention.

The permutation test: a simple way to test hypotheses.

Quantitative researchers can use permutation tests to conduct null hypothesis significance testing without resorting to complicated distribution theory. A permutation test can reach conclusions in hypothesis testing that are the same as those of better-known tests such as the t-test but is much easier to understand and implement. To introduce and explain permutation tests using two real examples of independent and dependent t-tests and their corresponding permutation tests. This article traces the history of permutation tests, explains the possible reason for their absence in textbooks and offers a simple example of their implementation. It provides simple code written in the R programming language to generate the null distributions and P -values for the permutation tests. Permutation tests do not require the strict model assumptions of t -tests and can be robust alternatives. Permutation tests are a useful addition to practitioners' research repertoire for testing hypotheses.

A Conceptual Framework for Quantifying the Robustness of a Regression-Based Causal Inference in Observational Study

The internal validity of a causal inference made based on an observational study is often subject to debate. The potential outcomes framework of causal inference stipulates that causal inference is essentially a missing data problem, and we follow this spirit to define the ideal sample as the combination of the observed data and the missing/counterfactual data for regression models. The robustness of a causal inference can be quantified by the probability of a robust inference for internal validity in regression, i.e., the PIVR, which is the probability of rejecting the null hypothesis again for the ideal sample provided the same null hypothesis has been already rejected for the observed sample. Drawing on the relationship between the PIVR and the mean counterfactual outcomes, we formalize a conceptual framework of quantifying the robustness of a regression-based causal inference based on a joint distribution about the mean counterfactual outcomes, holding the observed sample fixed. Interpretatively, the PIVR is the statistical power of the null hypothesis significance testing that is thought to be built on the ideal sample. We demonstrate the conceptual framework of quantifying the robustness of a regression-based causal inference with an empirical example.

Diagnosing the Misuse of the Bayes Factor in Applied Research

Hypothesis testing is often used for inference in the social sciences. In particular, null hypothesis significance testing (NHST) and its p value have been ubiquitous in published research for decades. Much more recently, null hypothesis Bayesian testing (NHBT) and its Bayes factor have also started to become more commonplace in applied research. Following preliminary work by Wong and colleagues, we investigated how, and to what extent, researchers misapply the Bayes factor in applied psychological research by means of a literature study. Based on a final sample of 167 articles, our results indicate that, not unlike NHST and the [Formula: see text] value, the use of NHBT and the Bayes factor also shows signs of misconceptions. We consider the root causes of the identified problems and provide suggestions to improve the current state of affairs. This article is aimed to assist researchers in drawing the best inferences possible while using NHBT and the Bayes factor in applied research.

A Utilitarian Perspective on Risk Quantification for Clinical Significance in Binary Outcomes.

Null hypothesis significance testing (NHST) in medical research is increasingly being supplemented by estimation statistics, focusing on effect sizes (ESs) and confidence intervals (CIs). This study evaluates the expression of ESs and CIs for binary outcomes. A utilitarian framework is proposed, emphasizing the number of beneficiaries and the impact level. To evaluate clinical significance, minimal clinically important risk difference (MCIRD) is proposed based on event magnitude (EM). Within this framework, risk difference (RD) is introduced as the primary measure. To assess the performance of RD, we compared its statistical power against other measures (risk ratio, RR; odds ratio, OR; Cohen's h) in individual study scenarios, and visual information conveyance in meta-analysis scenarios. RDs maintain statistical power in comparison to other measures in individual studies. They provide clarity on the true impact of clinical interventions without compromising statistical integrity. Meta-analytic results indicate that using RDs directly enhances transparency, uncovers heterogeneity, and addresses misaligned assumptions. This approach, by quantifying clinical effectiveness under a utilitarian perspective, facilitates the applicability of research to patient care and encourages shared decision-making. The study advocates for reporting baseline risks (BRs) with RDs and recommends a standardized presentation of these statistics. In a utilitarian perspective, adopting RD as the preferred ES can foster a transparent, patient-focused research ethos. This aids in accurately presenting the magnitude and variability of treatment effects, offering a new direction in methodology.

Philosophical Preconditions Guide Null Hypothesis Significance Testing in Empirical Psychology

To mathematize is closely related to the ideal of scientific objectivity, claiming bias-free results. In this respect, mathematization may substantiate the belief that significance testing could prove or disprove a hypothesis, since it is considered bias-free. However, this belief overlooks the fact that scientific objectivity is an epistemic virtue. Additionally, reducing human activity to numbers inevitably involves simplifying the real world, which includes uncertainty. Although statistics is valuable to psychology and science, it may be vulnerable to ontological reductionism.

The relationship between facial attractiveness and perceived guilt across types of crime.

Facial first impressions influence jurors in both laboratory experiments and real courtrooms. Often, more attractive defendants are perceived as less guilty and receive more lenient sentences. However, the type of crime under consideration, as well as the ecological validity of the stimuli presented, may determine the nature of this bias. Here, extending previous work, we considered three crime types (robbery, sexual assault, and murder) and utilised short video clips of male defendants, accompanied by real-world crime descriptions. Crucially, we varied attractiveness by presenting a large set of identities, in comparison with the typical use of one "high" and one "low" attractive face. Using null hypothesis significance testing, we found no evidence that either attractiveness or crime type influenced guilt perceptions. Taking a Bayesian perspective, our results provided some evidence that more attractive defendants were rated as less guilty of murder but more guilty of sexual assault, with no bias observed for robbery. Importantly, however, none of these effects had high certainty and all were small in size. By comparing the extremes of attractiveness, we again found inconclusive evidence of any attractiveness effects, with small differences in all cases. The implications for this departure from previous findings are discussed in terms of ecological validity and the need to consider attractiveness as a continuous rather than binary measure.

A Bayesian Analysis of a Cognitive-Behavioral Therapy Intervention for High-Risk People on Probation

This analysis employs a Bayesian framework to estimate the impact of a Cognitive-Behavioral Therapy (CBT) intervention on the recidivism of high-risk people under community supervision. The study relies on the reanalysis of experimental datal using a Bayesian logistic regression model. In doing so, new estimates of programmatic impact were produced using weakly informative Cauchy priors and the Hamiltonian Monte Carlo method. The Bayesian analysis indicated that CBT reduced the prevalence of new charges for total, non-violent, property, and drug crimes. However, the effectiveness of the CBT program varied meaningfully depending on the participant's age. The probability of the successful reduction of drug offenses was high only for younger individuals (<26 years old), while there was an impact on property offenses only for older individuals (>26 years old). In general, the probability of the successful reduction of new charges was higher for the older group of people on probation. Generally, this study demonstrates that Bayesian analysis can complement the more commonplace Null Hypothesis Significance Test (NHST) analysis in experimental research by providing practically useful probability information. Additionally, the specific findings of the reestimation support the principles of risk-needs responsivity and risk-stratified community supervision and align with related findings, though important differences emerge. In this case, the Bayesian estimations suggest that the effect of the intervention may vary for different types of crime depending on the age of the participants. This is informative for the development of evidence-based correctional policy and effective community supervision programming.

Statistically significant differences versus convincing evidence of real treatment effects: an analysis of the false positive risk for single-centre trials in anaesthesia

BackgroundThe American Statistical Association has highlighted problems with null hypothesis significance testing and outlined alternative approaches that may ‘supplement or even replace P-values’. One alternative is to report the false positive risk (FPR), which quantifies the chance the null hypothesis is true when the result is statistically significant. MethodsWe reviewed single-centre, randomised trials in 10 anaesthesia journals over 6 yr where differences in a primary binary outcome were statistically significant. We calculated a Bayes factor by two methods (Gunel, Kass). From the Bayes factor we calculated the FPR for different prior beliefs for a real treatment effect. Prior beliefs were quantified by assigning pretest probabilities to the null and alternative hypotheses. ResultsFor equal pretest probabilities of 0.5, the median (inter-quartile range [IQR]) FPR was 6% (1–22%) by the Gunel method and 6% (1–19%) by the Kass method. One in five trials had an FPR ≥20%. For trials reporting P-values 0.01–0.05, the median (IQR) FPR was 25% (16–30%) by the Gunel method and 20% (16–25%) by the Kass method. More than 90% of trials reporting P-values 0.01–0.05 required a pretest probability >0.5 to achieve an FPR of 5%. The median (IQR) difference in the FPR calculated by the two methods was 0% (0–2%). ConclusionsOur findings suggest that a substantial proportion of single-centre trials in anaesthesia reporting statistically significant differences provide limited evidence of real treatment effects, or, alternatively, required an implausibly high prior belief in a real treatment effect. Clinical trial registrationPROSPERO (CRD42023350783).

Establishing driving simulator validity: drawbacks of null-hypothesis significance testing when compared to equivalence tests and Bayes factors

The validity of driving simulators is a topic that has been addressed ever since their first use. In practice, a comparison of field data and simulator data is often used for validation purposes. A systematic literature search was conducted which identified large heterogeneity in both study designs and statistical methods used in research on simulator validity. Most of the included studies concluded simulator validity (thus equivalence of field data and simulator data) from non-significant results in Null-Hypothesis Significance Testing (NHST), which is a well-known yet somewhat controversial practice. In this work, the consequences of both study design choice and the choice of the statistical method are outlined and alternatives such as using Bayes factor hypothesis testing and equivalence tests are discussed and applied to a real data example. The present paper is intended to shed light on the influence of the statistical method on conclusions drawn from experimental data. For research on driving simulator validity, we highly encourage the use of equivalence tests or Bayes factor hypothesis tests, in particular when evidence of the data suggests that an effect might be absent.

Using multiple outcomes in intervention studies: improving power while controlling type I errors

Background The CONSORT guidelines for clinical trials recommend use of a single primary outcome, to guard against the raised risk of false positive findings when multiple measures are considered. It is, however, possible to include a suite of multiple outcomes in an intervention study, while controlling the familywise error rate, if the criterion for rejecting the null hypothesis specifies that N or more of the outcomes reach an agreed level of statistical significance, where N depends on the total number of outcome measures included in the study, and the correlation between them. Methods Simulations were run, using a conventional null-hypothesis significance testing approach with alpha set at .05, to explore the case when between 2 and 12 outcome measures are included to compare two groups, with average correlation between measures ranging from zero to .8, and true effect size ranging from 0 to .7. In step 1, a table is created giving the minimum N significant outcomes (MinNSig) that is required for a given set of outcome measures to control the familywise error rate at 5%. In step 2, data are simulated using MinNSig values for each set of correlated outcomes and the resulting proportion of significant results is computed for different sample sizes,correlations, and effect sizes. Results The Adjust NVar approach can achieve a more efficient trade-off between power and type I error rate than use of a single outcome when there are three or more moderately intercorrelated outcome variables. Conclusions Where it is feasible to have a suite of moderately correlated outcome measures, then this might be a more efficient approach than reliance on a single primary outcome measure in an intervention study. In effect, it builds in an internal replication to the study. This approach can also be used to evaluate published intervention studies.

Null regions: a unified conceptual framework for statistical inference.

Ruling out the possibility that there is absolutely no effect or association between variables may be a good first step, but it is rarely the ultimate goal of science. Yet that is the only inference provided by traditional null hypothesis significance testing (NHST), which has been a mainstay of many scientific fields. Reliance on NHST also makes it difficult to define what it means to replicate a finding, and leads to an uncomfortable quandary in which increasing precision in data reduces researchers' ability to perform theory falsification. To solve these problems, in recent years several alternatives to traditional NHST have been proposed. However, each new test is described using its own terminology and practiced in different fields. We describe a simple, unified framework for conceptualizing all these tests so that it is not necessary to learn them separately. Moreover, the framework allows researchers to conduct any of these tests by asking just one question: is the confidence interval entirely outside the null region(s)? This framework may also help researchers choose the test(s) that best answers their research question when simply ruling out 'no effect at all' is not enough.