Null Hypothesis Significance Testing Research Articles

Applying generalized funnel plots to help design statistical analyses

Researchers across many fields routinely analyze trial data using Null Hypothesis Significance Tests with zero null and p < 0.05. To promote thoughtful statistical testing, we propose a visualization tool that highlights practically meaningful effects when calculating sample sizes. The tool re-purposes and adapts funnel plots, originally developed for meta-analyses, after generalizing them to cater for meaningful effects. As with traditional sample size calculators, researchers must nominate anticipated effect sizes and variability alongside the desired power. The advantage of our tool is that it simultaneously presents sample sizes needed to adequately power tests for equivalence, for non-inferiority and for superiority, each considered at up to three alpha levels and in positive and negative directions. The tool thus encourages researchers at the design stage to think about the type and level of test in terms of their research goals, costs of errors, meaningful effect sizes and feasible sample sizes. An R-implementation of the tool is available on-line.

Why and how we should join the shift from significance testing to estimation.

A paradigm shift away from null hypothesis significance testing seems in progress. Based on simulations, we illustrate some of the underlying motivations. First, p‐values vary strongly from study to study, hence dichotomous inference using significance thresholds is usually unjustified. Second, ‘statistically significant’ results have overestimated effect sizes, a bias declining with increasing statistical power. Third, ‘statistically non‐significant’ results have underestimated effect sizes, and this bias gets stronger with higher statistical power. Fourth, the tested statistical hypotheses usually lack biological justification and are often uninformative. Despite these problems, a screen of 48 papers from the 2020 volume of the Journal of Evolutionary Biology exemplifies that significance testing is still used almost universally in evolutionary biology. All screened studies tested default null hypotheses of zero effect with the default significance threshold of p = 0.05, none presented a pre‐specified alternative hypothesis, pre‐study power calculation and the probability of ‘false negatives’ (beta error rate). The results sections of the papers presented 49 significance tests on average (median 23, range 0–390). Of 41 studies that contained verbal descriptions of a ‘statistically non‐significant’ result, 26 (63%) falsely claimed the absence of an effect. We conclude that studies in ecology and evolutionary biology are mostly exploratory and descriptive. We should thus shift from claiming to ‘test’ specific hypotheses statistically to describing and discussing many hypotheses (possible true effect sizes) that are most compatible with our data, given our statistical model. We already have the means for doing so, because we routinely present compatibility (‘confidence’) intervals covering these hypotheses.

The impact of GenMark Dx ePlex blood-culture identification on the treatment and outcomes of gram-positive bacteremia

Background: In the treatment of bloodstream infections, the identification of the causal pathogen, and the evaluation of its susceptibility to antibiotics, often serve as the rate-limiting steps of the patient’s hospital stay. The GenMark Dx ePlex blood culture identification gram-positive (BCID-GP) panel aims to alleviate this bottleneck, thereby reducing the risk of severe complications and the spread of resistance, using electrowetting technology to detect the most common causes of GP bacteremia (20 targets) and 4 antimicrobial resistance (AMR) genes. We hypothesized that implementation of the ePlex BCID-GP panel would improve antimicrobial choice and de-escalation where appropriate. Methods: A mixed blinded and unblinded study was conducted to assess the effect of the BCID-GP panel on the outcomes and antibiotic stewardship of GP bacteremic patients before ePlex results were made clinically available (before implementation, N = 73) and once they accompanied the standard-of-care work-up (after implementation, N = 82). Differences in time to different benchmarks between the 2 modalities and the effect on patient outcomes were analyzed using null-hypothesis significance testing. Results: During the study, the BCID-GP panel identified 63 (42%) Staphylococcus epidermidis isolates, 31 (21%) Staphylococcus spp, 24 (16%) Staphylococcus aureus isolates, 12 (8%) Streptococcus spp, and 7 (5%) Enterococcus spp, and results were similar in the pre- and postimplementation groups (P = .13). The panel saved an average of 32.0 ± 24.2 hours in pathogen identification over standard-of-care methods, with no statistical difference made by the clinical availability of the data (Table 1). In terms of susceptibility testing, the panel saved an average of 70.1 ± 58.2 hours but with less unity between the 2 cohorts (P = .005). Of the 66 cases with follow-up, identification via ePlex indicated an escalation of therapy in 20 (30%) and a narrowing of coverage in 31 (47%). In patients identified to have Staphylococcus aureus, BCID-GP could change antimicrobial therapy in 79%; the need for escalation of antibiotics was identified in 58% of cases. In patients with Staphylococcus epidermidis bacteremia, implementation of BCID-GP panel could have resulted in de-escalation of antimicrobial therapy in 67% of patients. The implementation of the BCID-GP panel was correlated with no significant change of in-hospital mortality (P = .72) but was correlated with a significantly decreased death-censored total length of stay (LOS) (P &lt; .001) and LOS after culture (P = .001). Conclusions: Our study has demonstrated that nonculture identification of bacteria and susceptibility can result in major improvements in antimicrobial therapy in patients, particularly those with contaminants identified.Funding: GenMark DXDisclosures: None

Current controversies: Null hypothesis significance testing.

Traditional null hypothesis significance testing (NHST) incorporating the critical level of significance of 0.05 has become the cornerstone of decision-making in health care, and nowhere less so than in obstetric and gynecological research. However, such practice is controversial. In particular, it was never intended for clinical significance to be inferred from statistical significance. The inference of clinical importance based on statistical significance (p < 0.05), and lack of clinical significance otherwise (p ≥ 0.05) represents misunderstanding of the original purpose of NHST. Furthermore, the limitations of NHST-sensitivity to sample size, plus type I and II errors-are frequently ignored. Therefore, decision-making based on NHST has the potential for recurrent false claims about the effectiveness of interventions or importance of exposure to risk factors, or dismissal of important ones. This commentary presents the history behind NHST along with the limitations that modern-day NHST presents, and suggests that a statistics reform regarding NHST be considered.

Statistical Inference of Prehistoric Demography from Frequency Distributions of Radiocarbon Dates: A Review and a Guide for the Perplexed

The last decade saw a rapid increase in the number of studies where time–frequency changes of radiocarbon dates have been used as a proxy for inferring past population dynamics. Although its universal and straightforward premise is appealing and undoubtedly offers some unique opportunities for research on long-term comparative demography, practical applications are far from trivial and riddled with issues pertaining to the very nature of the proxy under examination. Here I review the most common criticisms concerning the nature of radiocarbon time–frequency data as a demographic proxy, focusing on key statistical and inferential challenges. I then examine and compare recent methodological advances in the field by grouping them into three approaches: reconstructive, null-hypothesis significance testing, and model fitting. I will then conclude with some general recommendations for applying these techniques in archaeological and paleo-demographic research.

Waldian t tests: Sequential Bayesian t tests with controlled error probabilities.

Bayesian t tests have become increasingly popular alternatives to null-hypothesis significance testing (NHST) in psychological research. In contrast to NHST, they allow for the quantification of evidence in favor of the null hypothesis and for optional stopping. A major drawback of Bayesian t tests, however, is that error probabilities of statistical decisions remain uncontrolled. Previous approaches in the literature to remedy this problem require time-consuming simulations to calibrate decision thresholds. In this article, we propose a sequential probability ratio test that combines Bayesian t tests with simple decision criteria developed by Abraham Wald in 1947. We discuss this sequential procedure, which we call Waldian t test, in the context of three recently proposed specifications of Bayesian t tests. Waldian t tests preserve the key idea of Bayesian t tests by assuming a distribution for the effect size under the alternative hypothesis. At the same time, they control expected frequentist error probabilities, with the nominal Type I and Type II error probabilities serving as upper bounds to the actual expected error rates under the specified statistical models. Thus, Waldian t tests are fully justified from both a Bayesian and a frequentist point of view. We highlight the relationship between Bayesian and frequentist error probabilities and critically discuss the implications of conventional stopping criteria for sequential Bayesian t tests. Finally, we provide a user-friendly web application that implements the proposed procedure for interested researchers. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Does it work? Using a Meta-Impact score to examine global effects in quasi-experimental intervention studies.

The evaluation of applied psychological interventions in the workplace or elsewhere is challenging. Randomisation and matching are difficult to achieve and this often results in substantial heterogeneity within intervention and control groups. As a result, traditional comparison of group means using null hypothesis significance testing may mask effects experienced by some participants. Using longitudinal studies of coaching interventions designed to provide support for dyslexic employees, this study describes and evaluates a different approach using a Meta-Impact score. We offer a conceptual rationale for our method, illustrate how this score is calculated and analysed, and show how it highlights person-specific variations in how participants react and respond to interventions. We argue that Meta-Impact is an incremental supplement to traditional variable-centric group-wise comparisons and can more accurately demonstrate in practice the extent to which an intervention worked. Such methods are needed for applied research, where personalized intervention protocols may require impact analysis for policy, legal and ethical purposes, despite modest sample sizes.

Understanding loading patterns of construction waste hauling trucks: triangulation between big quantitative and informative qualitative data.

This research aims to understand the loading patterns of construction waste hauling trucks in Hong Kong and the factors shaping these patterns. It does so by triangulating the analytical results of big data collected from secondary sources and qualitative data from interviews. Firstly, based on the literature review and our engagement with the industry, four hypotheses on the nexus between "loading pattern" and the factors of (1) vehicle, (2) permitted gross vehicle weight, (3) commodity, and (4) ownership. Then, the hypotheses are tested with combined null hypothesis significance test and effect size measure using 13 million construction waste transportation records. Finally, the results are triangulated with interview data to empirically validate the nexus while providing sensible explanations to them. We find that the four hypotheses are all supported. Distinct loading patterns are presented by different types of (1) construction waste hauling trucks with different (2) permitted gross vehicle weights, (3) types of construction waste transported, and (4) ownership. These findings provide valuable evidence for more targeted interventions, e.g., introducing public policies or hauling operation optimization through the avoidance of excessive underloading or overloading.

False-positive and false-negative risks for individual multicentre trials in critical care.

In medical research, null hypothesis significance testing (NHST) is the dominant framework for statistical inference. NHST involves calculating P-values and confidence intervals to quantify the evidence against the null hypothesis of no effect. However, P-values and confidence intervals cannot tell us the probability that the hypothesis is true. In contrast, false-positive risk (FPR) and false-negative risk (FNR) are post-test probabilities concerning the truth of the hypothesis, that is to say, the probability a real effect exists. We calculated the FPR or FNR for 53 individual multicentre trials in critical care based on a pretest probability of 0.5 that the hypothesis was true. For trials reporting statistical significance, the FPR varied between 0.1% and 57.6%. For trials reporting non-significance, the FNR varied between 1.7% and 36.9%. Twenty-six of 47 trials (55.3%) reporting non-significance provided strong or very strong evidence in favour of the null hypothesis; the remaining trials provided limited evidence. There was no obvious relationship between the P-value and the FNR. The FPR and FNR showed marked variability, indicating that the probability of a real or absent treatment effect differed substantially between trials. Only one trial reporting statistical significance provided convincing evidence of a real treatment effect, and nearly half of all trials reporting non-significance provided limited evidence for the absence of a treatment effect. Our findings suggest that the quality of evidence from multicentre trials in critical care is highly variable.

Is the intervention as good as (or not substantially worse than) a comparator?

In the present issue, Mazzolari et al. (2022) present a persuasive case for the wider use of frequentist interval testing (for equivalence and non-inferiority) for interventional studies in exercise physiology and sport science. A persistent issue in the field is small, noisy, underpowered studies. With conventional null hypothesis significance tests for superiority, claims of no (zero) effect abound when P > 0.05. However, the confidence interval for the estimated effect would often show that biologically or clinically meaningful effects (both ‘positive’ and ‘negative’, benefit and harm) were compatible with the data and model and may not be ruled out statistically. If we fully embrace and discuss uncertainty in the estimation of intervention effects, then a more robust conclusion in such situations would be ‘get more data – more evidence needed’. The famous phrase ‘absence of evidence is not evidence of absence’ (Altman & Bland, 1995) articulates this problem elegantly and succinctly. Here, ‘absence of evidence’ means, conventionally, P > 0.05 for the mean intervention effect (intervention minus comparator). Since Altman and Bland's note, this issue has been revisited in many journal editorials in an attempt to improve the reporting of trial results (e.g., Alderson, 2004). Mazzolari et al. (2022) note that frequentist interval testing is applicable if the research question focuses on whether two interventions have similar efficacy (equivalence) or whether one intervention is not substantially worse than another (non-inferiority). These approaches are used frequently in biomedical research. For example, a new treatment might be developed that might cost less, be less burdensome encouraging greater compliance, or be associated with fewer adverse events than the accepted standard treatment. In such cases we might want to know whether the proposed new treatment worked as well as, or at least not substantially worse than, the standard treatment. The authors identify a range of potential applications of these approaches in the physiology and sport science research field. Frequentist interval testing provides the desired error control and may often be a better match to the research questions that researchers are really interested in answering – questions that conventional null hypothesis significance testing for superiority cannot address. So, how do physiology and sport science researchers apply these methods in their work? The article by Mazzolari et al. (2022) is a tour de force that covers all of the key issues, such as how to establish the equivalence or non-inferiority margins, and sample size planning for these designs. The article references provide an excellent reading list for those wishing to delve deeper and increase their understanding. At the very least, this contribution will increase awareness and facilitate improved critical reading of published research. Those wishing to apply the methods in the analysis of their own data will benefit from the worked examples, including the accompanying workbook with code for two popular statistics software packages. Anyone considering an equivalence or non-inferiority study should seek expert statistics input from the outset. This is sage advice for any study, but perhaps especially for these designs wherein there are many pitfalls for the unwary. Many, if not all, readers will be familiar with the Consolidated Standards of Reporting Trials (CONSORT) statement, providing a checklist for the complete and transparent reporting of parallel-arm (independent groups) randomized trials. An extension to this statement is available for equivalence and non-inferiority studies (Piaggio et al., 2012). In summary, this article is essential reading for consumers and doers of research in physiology and sport science, and I commend it to the journal readership. None.

The that-trace effect and island boundary-gap effect are the same: Demonstrating equivalence with null hypothesis significance testing and psychometrics

This paper demonstrates a novel approach in experimental syntax, leveraging psychometric methods to resolve a decades-old puzzle.&amp;nbsp; Specifically, gaps in subject position are more acceptable than gaps in object position in non-islands, while the reverse is true in islands (the Island Boundary-Gap Effect).&amp;nbsp; Attempts at explaining this asymmetry generally attribute it to a violation of the same constraint that renders gaps unacceptable after the overt complementizer `that' (the That-Trace Effect).&amp;nbsp; However, the two effects involve distinct syntactic structures, and there is no a priori reason to believe they are the same beyond the elegance of a unified account.&amp;nbsp; One limitation has been the difficulty of testing for equivalence in the Null Hypothesis Significance Testing framework: when two constructs behave similarly, it generally constitutes an uninterpretable null result. Experiments 1 and 2 use standard experimental methods to circumvent this problem, but ultimately provide evidence that is at best just consistent with equivalence.&amp;nbsp; Experiment 3 demonstrates a novel approach which shows that individual differences in the That-Trace Effect correlate with individual differences in the Island Boundary-Gap Effect, after removing correlated variance from carefully-chosen controls.&amp;nbsp; This psychometric approach provides positive evidence that the two effects do indeed derive from the same underlying phenomenon.

Feasibility of training the dorsolateral prefrontal-striatal network by real-time fMRI neurofeedback

Real-time fMRI neurofeedback (rt-fMRI NF) is a promising non-invasive technique that enables volitional control of usually covert brain processes. While most rt-fMRI NF studies so far have demonstrated the ability of the method to evoke changes in brain activity and improve symptoms of mental disorders, a recently evolving field is network-based functional connectivity (FC) rt-fMRI NF. However, FC rt-fMRI NF has methodological challenges such as respirational artefacts that could potentially bias the training if not controlled. In this randomized, double-blind, yoke-controlled, pre-registered FC rt-fMRI NF study with healthy participants (N = 40) studied over three training days, we tested the feasibility of an FC rt-fMRI NF approach with online global signal regression (GSR) to control for physiological artefacts for up-regulation of connectivity in the dorsolateral prefrontal-striatal network. While our pre-registered null hypothesis significance tests failed to reach criterion, we estimated the FC training effect at a medium effect size at the end of the third training day after rigorous control of physiological artefacts in the offline data. This hints at the potential of FC rt-fMRI NF for the development of innovative transdiagnostic circuit-specific interventional approaches for mental disorders and the effect should now be confirmed in a well-powered study.

Unifying the analysis of continuous and categorical measures of weight loss and incorporating group effect: a secondary re-analysis of a large cluster randomized clinical trial using Bayesian approach

BackgroundAlthough frequentist paradigm has been the predominant approach to clinical studies for decades, some limitations associated with the frequentist null hypothesis significance testing have been recognized. Bayesian approaches can provide additional insights into data interpretation and inference by deriving posterior distributions of model parameters reflecting the clinical interest. In this article, we sought to demonstrate how Bayesian approaches can improve the data interpretation by reanalyzing the Rural Engagement in Primary Care for Optimizing Weight Reduction (REPOWER).MethodsREPOWER is a cluster randomized clinical trial comparing three care delivery models: in-clinic individual visits, in-clinic group visits, and phone-based group visits. The primary endpoint was weight loss at 24 months and the secondary endpoints included the proportions of achieving 5 and 10% weight loss at 24 months. We reanalyzed the data using a three-level Bayesian hierarchical model. The posterior distributions of weight loss at 24 months for each arm were obtained using Hamiltonian Monte Carlo. We then estimated the probability of having a higher weight loss and the probability of having greater proportion achieving 5 and 10% weight loss between groups. Additionally, a four-level hierarchical model was used to assess the partially nested intervention group effect which was not investigated in the original REPOWER analyses.ResultsThe Bayesian analyses estimated 99.5% probability that in-clinic group visits, compared with in-clinic individual visits, resulted in a higher percent weight loss (posterior mean difference: 1.8%[95% CrI: 0.5,3.2%]), a greater probability of achieving 5% threshold (posterior mean difference: 9.2% [95% CrI: 2.4, 16.0%]) and 10% threshold (posterior mean difference: 6.6% [95% CrI: 1.7, 11.5%]). The phone-based group visits had similar result. We also concluded that including intervention group did not impact model fit significantly.ConclusionsWe unified the analyses of continuous (the primary endpoint) and categorical measures (the secondary endpoints) of weight loss with one single Bayesian hierarchical model. This approach gained statistical power for the dichotomized endpoints by leveraging the information in the continuous data. Furthermore, the Bayesian analysis enabled additional insights into data interpretation and inference by providing posterior distributions for parameters of interest and posterior probabilities of different hypotheses that were not available with the frequentist approach.Trial registrationClinicalTrials.gov Identifier NCT02456636; date of registry: May 28, 2015.

A Review of Bayesian Hypothesis Testing and Its Practical Implementations.

We discuss hypothesis testing and compare different theories in light of observed or experimental data as fundamental endeavors in the sciences. Issues associated with the p-value approach and null hypothesis significance testing are reviewed, and the Bayesian alternative based on the Bayes factor is introduced, along with a review of computational methods and sensitivity related to prior distributions. We demonstrate how Bayesian testing can be practically implemented in several examples, such as the t-test, two-sample comparisons, linear mixed models, and Poisson mixed models by using existing software. Caveats and potential problems associated with Bayesian testing are also discussed. We aim to inform researchers in the many fields where Bayesian testing is not in common use of a well-developed alternative to null hypothesis significance testing and to demonstrate its standard implementation.

Results Blind Science Publishing and a Decision-Theoretic Approach to Publishing

In this paper, I revisit my earlier proposal for Results Blind Publishing (RBP) and have added some new perspectives and qualifications regarding it. RBP is a suggestion that research journals decide on publication of submitted manuscripts based on reviewing only their Introduction section (which suggests the substantive importance of the research question addressed by the study) and Methods section (which suggests how likely the study validly answers that question), as a means of avoiding publication bias based on what the results are claimed to be, a bias that exacerbates replicability problems. I differentiate the separate questions of: (1) what should be the criteria for a positive as opposed to null research finding versus (2) what should be the criteria for publication of manuscripts. I believe the conflation of these two different questions fuels some of the controversies and confusion concerning null hypothesis significance testing and similar issues. I also compare the pros and cons of RBP versus the practice of preregistering studies. Further, I cite a potentially serious problem with RBP and suggest a fix-up involving a decision theoretic approach to manuscript publication.

Effects of Sentence Output Tasks on EFL Vocabulary Learning from a Bayesian Perspective

This article builds on the growing line of inquiry into the relative effectiveness of different tasks for vocabulary learning. Specifically, the study compares the efficacy of several sentence output tasks in EFL vocabulary learning through reading. To this end, evidence was weighed for one hypothesis over the alternative from a Bayesian perspective rather than in light of the commonly used null hypothesis significance testing (NHST), which depends heavily on the <i>p</i> values for statistical conclusions. Ninety-one EFL learners were randomly assigned to one of three word-focused sentence output tasks (i.e., L2-L1 translating, paraphrasing and writing), and were subsequently tested on their initial learning and retention of newly-encountered EFL words. Both Bayes factor analysis and Bayesian parameter estimation were employed to find evidence for task effects. With respect to initial word learning, moderate evidence was in favor of no difference between translating and paraphrasing, whilst weak evidence in favor of no difference between translating and writing as well as between paraphrasing and writing. For pedagogical purposes, no good evidence was for or against task effects. Regarding word retention, moderate evidence supported no task difference, and all the tasks fared equally well pedagogically. The results partially support the involvement load hypothesis and are discussed in terms of task difficulty, context generation and semantic elaboration.

The state of the art of hypothesis testing in the social sciences

Over many decades, one seemingly fatal critique after another has been launched against the use of social sciences' dominant practice of null-hypothesis significance testing, also known as NHST. In the last decade, we have witnessed a further upsurge in this critique, associated with suggestions as to how to conduct quantitative empirical work differently. In this research note, we closely review 148 articles that report findings from quantitative empirical studies published in recent issues of 18 top journals in six prominent disciplines in the social sciences: business, economics, political science, psychology, public administration, and sociology. We conclude that NHST is still overly dominant in all six disciplines, although many economics' studies tend to refer to ‘estimation precision’ rather than ‘statistical significance’. Additionally, we identify a few examples of novel practices that may point the way to change.

Reverse-Bayes methods for evidence assessment and research synthesis.

It is now widely accepted that the standard inferential toolkit used by the scientific research community—null‐hypothesis significance testing (NHST)—is not fit for purpose. Yet despite the threat posed to the scientific enterprise, there is no agreement concerning alternative approaches for evidence assessment. This lack of consensus reflects long‐standing issues concerning Bayesian methods, the principal alternative to NHST. We report on recent work that builds on an approach to inference put forward over 70 years ago to address the well‐known “Problem of Priors” in Bayesian analysis, by reversing the conventional prior‐likelihood‐posterior (“forward”) use of Bayes' theorem. Such Reverse‐Bayes analysis allows priors to be deduced from the likelihood by requiring that the posterior achieve a specified level of credibility. We summarise the technical underpinning of this approach, and show how it opens up new approaches to common inferential challenges, such as assessing the credibility of scientific findings, setting them in appropriate context, estimating the probability of successful replications, and extracting more insight from NHST while reducing the risk of misinterpretation. We argue that Reverse‐Bayes methods have a key role to play in making Bayesian methods more accessible and attractive for evidence assessment and research synthesis. As a running example we consider a recently published meta‐analysis from several randomised controlled trials (RCTs) investigating the association between corticosteroids and mortality in hospitalised patients with COVID‐19.

Bland–Altman Limits of Agreement from a Bayesian and Frequentist Perspective

Bland–Altman agreement analysis has gained widespread application across disciplines, last but not least in health sciences, since its inception in the 1980s. Bayesian analysis has been on the rise due to increased computational power over time, and Alari, Kim, and Wand have put Bland–Altman Limits of Agreement in a Bayesian framework (Meas. Phys. Educ. Exerc. Sci. 2021, 25, 137–148). We contrasted the prediction of a single future observation and the estimation of the Limits of Agreement from the frequentist and a Bayesian perspective by analyzing interrater data of two sequentially conducted, preclinical studies. The estimation of the Limits of Agreement θ1 and θ2 has wider applicability than the prediction of single future differences. While a frequentist confidence interval represents a range of nonrejectable values for null hypothesis significance testing of H0: θ1 ≤ −δ or θ2 ≥ δ against H1: θ1 &gt; −δ and θ2 &lt; δ, with a predefined benchmark value δ, Bayesian analysis allows for direct interpretation of both the posterior probability of the alternative hypothesis and the likelihood of parameter values. We discuss group-sequential testing and nonparametric alternatives briefly. Frequentist simplicity does not beat Bayesian interpretability due to improved computational resources, but the elicitation and implementation of prior information demand caution. Accounting for clustered data (e.g., repeated measurements per subject) is well-established in frequentist, but not yet in Bayesian Bland–Altman analysis.

Misinterpretations of Null Hypothesis Significance Testing Results Near the P-Value Threshold in the Neurosurgical Literature

Although statistical errors have been shown to be prevalent in the medical literature, there has been a paucity of studies focusing on the prevalence of statistical errors in neurosurgery. We examined the contemporary neurosurgical literature for a common statistical mistake, namely, misinterpretations of null hypothesis significance testing results near the P-value threshold by conveying the idea of a trend. PubMed/Medline was used to identify all articles published in 6 major neurosurgical journals between 2000 and 2020. The abstracts of these articles were extracted and scrutinized to determine when the word "trend" was used to express near statistical significance. This study included a total of 45,244 articles. Of those, 461 (1.02% [95% confidence interval, 0.86%-1.18%]) employed the word "trend" to indicate near statistical significance, for a total of 3.8 (95% confidence interval, 2.93-4.75) errors per issue per year. The error under study occurred more frequently in Journal of Neurosurgery: Spine and less frequently in Acta Neurochirurgica (P= 0.007). On an annual basis, there was no linear correlation between the total number of articles published per journal and the number of such errors in that journal (r= 0.34, P=0.01). Misinterpretations of null hypothesis significance testing results near the P-value threshold are present in at least 1% of the neurosurgical literature. While we believe that most statistical errors in medicine in general and neurosurgery in particular may be unintentional in nature, additional measures should be put in place to prevent the subsequent adoption of such undesirable methodological practices among future researchers.