Traditional Null Hypothesis Significance Testing Research Articles

Frequentist, Bayesian Analysis and Complementary Statistical Tools for Geriatric and Rehabilitation Fields: Are Traditional Null-Hypothesis Significance Testing Methods Sufficient?

Null hypothesis significant testing (NHST) is the dominant statistical approach in the geriatric and rehabilitation fields. However, NHST is routinely misunderstood or misused. In this case, the findings from clinical trials would be taken as evidence of no effect, when in fact, a clinically relevant question may have a "non-significant" p-value. Conversely, findings are considered clinically relevant when significant differences are observed between groups. To assume that p-value is not an exclusive indicator of an association or the existence of an effect, researchers should be encouraged to report other statistical analysis approaches as Bayesian analysis and complementary statistical tools alongside the p-value (eg, effect size, confidence intervals, minimal clinically important difference, and magnitude-based inference) to improve interpretation of the findings of clinical trials by presenting a more efficient and comprehensive analysis. However, the focus on Bayesian analysis and secondary statistical analyses does not mean that NHST is less important. Only that, to observe a real intervention effect, researchers should use a combination of secondary statistical analyses in conjunction with NHST or Bayesian statistical analysis to reveal what p-values cannot show in the geriatric and rehabilitation studies (eg, the clinical importance of 1kg increase in handgrip strength in the intervention group of long-lived older adults compared to a control group). This paper provides potential insights for improving the interpretation of scientific data in rehabilitation and geriatric fields by utilizing Bayesian and secondary statistical analyses to better scrutinize the results of clinical trials where a p-value alone may not be appropriate to determine the efficacy of an intervention.

Null regions: a unified conceptual framework for statistical inference.

Ruling out the possibility that there is absolutely no effect or association between variables may be a good first step, but it is rarely the ultimate goal of science. Yet that is the only inference provided by traditional null hypothesis significance testing (NHST), which has been a mainstay of many scientific fields. Reliance on NHST also makes it difficult to define what it means to replicate a finding, and leads to an uncomfortable quandary in which increasing precision in data reduces researchers' ability to perform theory falsification. To solve these problems, in recent years several alternatives to traditional NHST have been proposed. However, each new test is described using its own terminology and practiced in different fields. We describe a simple, unified framework for conceptualizing all these tests so that it is not necessary to learn them separately. Moreover, the framework allows researchers to conduct any of these tests by asking just one question: is the confidence interval entirely outside the null region(s)? This framework may also help researchers choose the test(s) that best answers their research question when simply ruling out 'no effect at all' is not enough.

Introduction to Bayesian Analyses for Clinical Research.

Bayesian analyses are becoming more popular as a means of analyzing data, yet the Bayesian approach is novel to many members of the broad clinical audience. While Bayesian analyses are foundational to anesthesia pharmacokinetic/pharmacodynamic modeling, they also can be used for analyzing data from clinical trials or observational studies. The traditional null hypothesis significance testing (frequentist) approach uses only the data collected from the current study to make inferences. On the other hand, the Bayesian approach quantifies the external information or expert knowledge and combines the external information with the study data, then makes inference from this combined information. We introduce to the clinical and translational science researcher what it means to do Bayesian statistics, why a researcher would choose to perform their analyses using the Bayesian approach, when it would be advantageous to use a Bayesian instead of a frequentist approach, and how Bayesian analyses and interpretations differ from the more traditional frequentist methods. Throughout this paper, we use various pain- and anesthesia-related examples to highlight the ideas and statistical concepts that should be relatable to other areas of research as well.

Working Memory Training for Children Using the Adaptive, Self-Select, and Stepwise Approaches to Setting the Difficulty Level of Training Activities: Protocol for a Randomized Controlled Trial.

A common yet untested assumption of cognitive training in children is that activities should be adaptive, with difficulty adjusted to the individual's performance in order to maximize improvements on untrained tasks (known as transfer). Working memory training provides the ideal testbed to systematically examine this assumption as it is one of the most widely studied domains in the cognitive training literature, and is critical for children's learning, including following instructions and reasoning. This trial aimed to examine children's outcomes of working memory training using adaptive, self-select (child selects difficulty level), and stepwise (difficulty level increases incrementally) approaches to setting the difficulty of training activities compared to an active control condition immediately and 6-month postintervention. While the aim is exploratory, we hypothesized that children allocated to a working memory training condition would show greater improvements: (1) on near transfer measures compared to intermediate and far transfer measures and (2) immediately postintervention compared to 6-month postintervention. This double-blinded, active-controlled, parallel-group randomized trial aimed to recruit 128 children aged 7 to 11 years from 1 metropolitan primary school in Melbourne, Australia. Following baseline testing, children were randomized into 1 of 4 conditions: adaptive, self-select, or stepwise working memory training, or active control. An experimental intervention embedded in Minecraft was developed for teachers to deliver in class over 2 consecutive weeks (10 × 20-minute sessions). The working memory training comprised 2 training activities with processing demands similar to daily activities: backward span and following instructions. The control comprised creative activities. Pre- and postintervention, children completed a set of working memory tests (near and intermediate transfer) and the Raven's Standard Progressive Matrices (far transfer) to determine training outcomes, as well as motivation questionnaires to determine if motivations toward learning and the intervention were similar across conditions. Caregivers completed the ADHD-Rating Scale-5 to measure their child's attention (far transfer). Statistical analysis will include traditional null hypothesis significance testing and Bayesian methods to quantify evidence for both the null and alternative hypotheses. Data collection concluded in December 2022. Data are currently being processed and analyzed. This trial will determine whether the adaptive approach to setting the difficulty of training activities maximizes cognitive training outcomes for children. This trial has several strengths: it adopts best practices for cognitive training studies (design, methods, and analysis plan); uses a range of measures to detect discrete levels of transfer; has a 6-month postintervention assessment; is appropriately powered; and uses an experimental working memory training intervention based on our current understanding of the cognitive mechanisms of training. Findings will inform future research and design of cognitive training interventions and highlight the value of the evidence-based principles of cognitive training. Australian New Zealand Clinical Trials Registry, ACTRN12621000990820; https://www.anzctr.org.au/ACTRN12621000990820.aspx. DERR1-10.2196/47496.

Reducing the false discovery rate of preclinical animal research with Bayesian statistical decision criteria.

The success of preclinical research hinges on exploratory and confirmatory animal studies. Traditional null hypothesis significance testing is a common approach to eliminate the chaff from a collection of drugs, so that only the most promising treatments are funneled through to clinical research phases. Balancing the number of false discoveries and false omissions is an important aspect to consider during this process. In this paper, we compare several preclinical research pipelines, either based on null hypothesis significance testing or based on Bayesian statistical decision criteria. We build on a recently published large-scale meta-analysis of reported effect sizes in preclinical animal research and elicit a non-informative prior distribution under which both approaches are compared. After correcting for publication bias and shrinkage of effect sizes in replication studies, simulations show that (i) a shift towards statistical approaches which explicitly incorporate the minimum clinically important difference reduces the false discovery rate of frequentist approaches and (ii) a shift towards Bayesian statistical decision criteria can improve the reliability of preclinical animal research by reducing the number of false-positive findings. It is shown that these benefits hold while keeping the number of experimental units low which are required for a confirmatory follow-up study. Results show that Bayesian statistical decision criteria can help in improving the reliability of preclinical animal research and should be considered more frequently in practice.

Stochastic process computational modeling for learning research

The goal of our research was to compare and systematize several approaches to non-parametric null hypothesis significance testing using computer-based statistical modeling. For teaching purposes, a statistical model for simulation of null hypothesis significance testing was created. The results were analyzed using Fisher's angular transformation, Chi-square, Mann-Whitney, and Fisher's exact tests. Appropriate software was created, allowing us to recommend new illustrative materials for expressing the limitations of the tests that were examined. Learning investigations as a technique of comprehending inductive statistics has been proposed, based on the fact that modern personal computers can run simulations in a reasonable amount of time with great precision. The collected results revealed that the most often used non-parametric tests for small samples have low power. Traditional null hypothesis significance testing does not allow students to analyze test power because the true differences between samples are unknown. As a result, in Ukrainian statistical education, including PhD studies, the emphasis must shift away from null hypothesis significance testing and toward statistical modeling as a modern and practical approach of establishing scientific hypotheses. This finding is supported by scientific papers and the American Statistical Association's recommendation.

Current controversies: Null hypothesis significance testing.

Traditional null hypothesis significance testing (NHST) incorporating the critical level of significance of 0.05 has become the cornerstone of decision-making in health care, and nowhere less so than in obstetric and gynecological research. However, such practice is controversial. In particular, it was never intended for clinical significance to be inferred from statistical significance. The inference of clinical importance based on statistical significance (p < 0.05), and lack of clinical significance otherwise (p ≥ 0.05) represents misunderstanding of the original purpose of NHST. Furthermore, the limitations of NHST-sensitivity to sample size, plus type I and II errors-are frequently ignored. Therefore, decision-making based on NHST has the potential for recurrent false claims about the effectiveness of interventions or importance of exposure to risk factors, or dismissal of important ones. This commentary presents the history behind NHST along with the limitations that modern-day NHST presents, and suggests that a statistics reform regarding NHST be considered.

Monte Carlo Analyses for Single-Case Experimental Designs: An Untapped Resource for Applied Behavioral Researchers and Practitioners

Group-based experimental designs are an outgrowth of the logic of null-hypothesis significance testing and thus, statistical tests are often considered inappropriate for single-case experimental designs. Behavior analysts have recently been more supportive of efforts to include appropriate statistical analysis techniques to evaluate single-case experimental design data. One way that behavior analysts can incorporate statistical analyses into their practices with single-case experimental designs is to use Monte Carlo analyses. These analyses compare experimentally obtained behavioral data to simulated samples of behavioral data to determine the likelihood that the experimentally obtained results occurred due to chance (i.e., a p value). Monte Carlo analyses are more in line with behavior analytic principles than traditional null-hypothesis significance testing. We present an open-source Monte Carlo tool, created in shiny, for behavior analysts who want to use Monte Carlo analyses in addition as part of their data analysis.

How to Choose between Different Bayesian Posterior Indices for Hypothesis Testing in Practice

Hypothesis testing is an essential statistical method in experimental psychology and the cognitive sciences. The problems of traditional null hypothesis significance testing (NHST) have been discussed widely, and among the proposed solutions to the replication problems caused by the inappropriate use of significance tests and p-values is a shift toward Bayesian data analysis. However, Bayesian hypothesis testing is concerned with various posterior indices for significance and the size of an effect. This complicates Bayesian hypothesis testing in practice, as the availability of multiple Bayesian alternatives to the traditional p-value causes confusion which one to select and why. In this paper, various Bayesian posterior indices which have been proposed in the literature are compared and their benefits and limitations are discussed. The comparison shows that conceptually not all proposed Bayesian alternatives to NHST and p-values are beneficial, and the usefulness of some indices strongly depends on the study design and research goal. However, the comparison also reveals that there exist at least two candidates among the available Bayesian posterior indices which have appealing theoretical properties and are widely underused in the cognitive sciences.

A Smartphone App to Support Sedentary Behavior Change by Visualizing Personal Mobility Patterns and Action Planning (SedVis): Development and Pilot Study.

BackgroundProlonged sedentary behavior is related to a number of risk factors for chronic diseases. Given the high prevalence of sedentary behavior in daily life, simple yet practical solutions for behavior change are needed to avoid detrimental health effects.ObjectiveThe mobile app SedVis was developed based on the health action process approach. The app provides personal mobility pattern visualization (for both physical activity and sedentary behavior) and action planning for sedentary behavior change. The primary aim of the study is to investigate the effect of mobility pattern visualization on users’ action planning for changing their sedentary behavior. The secondary aim is to evaluate user engagement with the visualization and user experience of the app.MethodsA 3-week user study was conducted with 16 participants who had the motivation to reduce their sedentary behavior. Participants were allocated to either an active control group (n=8) or an intervention group (n=8). In the 1-week baseline period, none of the participants had access to the functions in the app. In the following 2-week intervention period, only the intervention group was given access to the visualizations, whereas both groups were asked to make action plans every day and reduce their sedentary behavior. Participants’ sedentary behavior was estimated based on the sensor data of their smartphones, and their action plans and interaction with the app were also recorded by the app. Participants’ intention to change their sedentary behavior and user experience of the app were assessed using questionnaires.ResultsThe data were analyzed using both traditional null hypothesis significance testing (NHST) and Bayesian statistics. The results suggested that the visualizations in SedVis had no effect on the participants’ action planning according to both the NHST and Bayesian statistics. The intervention involving visualizations and action planning in SedVis had a positive effect on reducing participants’ sedentary hours, with weak evidence according to Bayesian statistics (Bayes factor, BF+0=1.92; median 0.52; 95% CI 0.04-1.25), whereas no change in sedentary time was more likely in the active control condition (BF+0=0.28; median 0.18; 95% CI 0.01-0.64). Furthermore, Bayesian analysis weakly suggested that the more frequently the users checked the app, the more likely they were to reduce their sedentary behavior (BF−0=1.49; r=−0.50).ConclusionsUsing a smartphone app to collect data on users’ mobility patterns and provide real-time feedback using visualizations may be a promising method to induce changes in sedentary behavior and may be more effective than action planning alone. Replications with larger samples are needed to confirm these findings.

Revisiting potential associations between brain morphology, fear acquisition and extinction through new data and a literature review

Inter-individual differences in defensive responding are widely established but their morphological correlates in humans have not been investigated exhaustively. Previous studies reported associations with cortical thickness of the dorsal anterior cingulate cortex, insula and medial orbitofrontal cortex as well as amygdala volume in fear conditioning studies. However, these associations are partly inconsistent and often derived from small samples. The current study aimed to replicate previously reported associations between physiological and subjective measures of fear acquisition and extinction and brain morphology. Structural magnetic resonance imaging was performed on 107 healthy adults who completed a differential cued fear conditioning paradigm with 24 h delayed extinction while skin conductance response (SCR) and fear ratings were recorded. Cortical thickness and subcortical volume were obtained using the software Freesurfer. Results obtained by traditional null hypothesis significance testing and Bayesians statistics do not support structural brain-behavior relationships: Neither differential SCR nor fear ratings during fear acquisition or extinction training could be predicted by cortical thickness or subcortical volume in regions previously reported. In summary, the current pre-registered study does not corroborate associations between brain morphology and inter-individual differences in defensive responding but differences in experimental design and analyses approaches compared to previous work should be acknowledged.

Assessing background particulate contamination in an historic building – surface lead loading and contamination

ABSTRACT Investigation of suspect surface contamination in a building may require comparative sampling across different zones to provide meaningful information with regard to contaminant sources, pathways and/or extent of dispersal. However, evaluation of the data using traditional null hypothesis significance testing (NHST) based upon the mean may result in misleading inference when encountering erratic distributions typical of environmental contaminant data. Sampling data (n = 90) for lead content in surface dust collected throughout a historic building with suspect contamination from uncontrolled disturbance to lead coatings were evaluated using traditional NHST and randomization/permutation inference; the latter metric was the maximum difference in frequency of detection (Δfd max), to directly calculate the probability of the observed differences. In the examples for lead in surface dust presented herein, areas with “lower” mean concentration and/or no significant difference via NHST actually represented “greater contamination,” as Δfd max indicated a greater probability of encountering lead at higher concentrations. Resulting conclusions with regard to sources and pathways contradicted those generated from traditional NHST, and underscore the need to recognize differences in applicability of different inference approaches, depending upon the distribution of the data and the particular problem. This is particularly relevant for forensic purposes. Implications The use of permutation/randomization inference to gain insight into sources and pathways of contamination may be more appropriate than the conventional Neyman/Pearson (N/P) logic in negative hypothesis significance testing (NHST). This suggests a broader understanding by environmental professionals of the assumptions and limitations of NHST and alternative inference such as through permutation/randomization is warranted.

An Introduction to Calculating Bayes Factors in JASP for Speech, Language, and Hearing Research.

Purpose Evidence-based data analysis methods are important in clinical research fields, including speech-language pathology and audiology. Although commonly used, null hypothesis significance testing (NHST) has several limitations with regard to the conclusions that can be drawn from results, particularly nonsignificant findings. Bayes factors (BFs) can be used to complement NHST and quantify the strength of evidence in favor of 1 hypothesis over another, given the data: commonly, either the alternate hypothesis over the null or the null hypothesis over the alternate. This article provides an introduction to BFs through JASP, a free, open-source, graphics-based statistics package that allows researchers to easily conduct both NHST and Bayesian analyses in a clear and reproducible manner. Method and Results Both traditional NHST analyses and Bayesian equivalents for correlations, t tests, and analyses of variance were conducted in JASP using simulated data, with explanations of analysis options, statistical output, and figures provided. These examples also demonstrate what NHST and BFs can and cannot infer about a data set. Additionally, BFs were calculated from the summary statistics of published nonsignificant results to illustrate how JASP may be useful to consumers of research who only have access to statistics provided in a published study. Conclusions Bayesian analyses are underutilized in speech, language, and hearing research. By complementing traditional NHST analyses with BFs, researchers can directly test for and quantify the strength of evidence during hypothesis testing, thereby drawing stronger conclusions from their research and providing more relevant information for clinicians and researchers in the field.

Power and Trust Dynamics of Sexual Violence: A Textual Analysis of Nassar Victim Impact Statements and #MeToo Disclosures on Twitter

Employing a natural language processing approach, we analyzed textual content derived from publicly-available athlete victim impact statements (VIS) from the Larry Nassar trial (N = 111) to examine psychosocial responses to sexual violence. To explore potential differences in a non-sports context, we conducted similar analyses on a sample of #MeToo tweets (N = 45,848). Our research focused on the semantic content of VIS, including positive and negative affect, power and trust dynamics, well-being, and post-traumatic growth. We hypothesized that athletes’ reactions to sexual violence would be more likely to contain language related to power and trust. Traditional null-hypothesis significance testing and network analyses were used to identify the psychosocial indicators unique to sexual violence disclosures in a sports context. Results indicated differential use of language related to negative affect, trust, power dynamics, and post-traumatic growth in sports versus non-sports contexts. We discuss clinical, practical, and policy-based implications for risk reduction and intervention.

Book review: ‘Statistics and Experimental Design for Psychologists: A model comparison approach’ by Rory Allen

This article reviews a new statistics textbook by Rory Allen. The book teaches the fundamentals of statistics and experimental design at an undergraduate psychology level. The material is taught from a model comparison perspective, which Allen argues to be more intuitive than traditional null hypothesis significance testing (NHST) approaches. A key feature of the book is an elegant and novel geometric formulation of Fisher’s F, the test statistic in analysis of variance (ANOVA). Allen’s contribution should provide a useful reference for psychology students, especially if future editions incorporate a wider range of statistical methods.

Exploring Small, Confirming Big: An alternative system to The New Statistics for advancing cumulative and replicable psychological research

While outlining his vision of The New Statistics, Cumming (2014) proposes that a more rigorous and cumulative psychological science will be built, in part, by having psychologists abandon traditional null-hypothesis significance testing (NHST) approaches, and conducting small-scale meta-analyses on their data whenever possible. In the present paper, I propose an alternative system for conducting rigorous and replicable psychological investigations, which I describe as Exploring Small, Confirming Big. I begin with a critical evaluation of the merits of NHST and small-scale meta-analyses, and argue that NHST does have a valuable role in the scientific process, whereas small-scale meta-analyses will do little to advance a cumulative science. I then present an overview of an alternative system for producing cumulative and replicable psychological research: Exploring Small, Confirming Big. It involves a two-step process to psychological research, consisting of (1) small N investigation(s), in which psychologists use NHST to develop exploratory models; and (2) strong, confirmatory tests of exploratory models, by analyzing new and/or existing large N datasets with variables that capture the effect(s) of interest from the Exploring Small stage. I conclude by discussing several anticipated benefits and challenges of adopting the Exploring Small, Confirming Big approach.

An examination of the equivalency of self-report measures obtained from crowdsourced versus undergraduate student samples.

Increasingly, researchers have begun to explore the potential of the Internet to reach beyond the traditional undergraduate sample. In the present study, we sought to compare the data obtained from a conventional undergraduate college-student sample to data collected via two online survey recruitment platforms. In order to examine whether the data sampled from the three populations were equivalent, we conducted a test of equivalency using inferential confidence intervals-an approach that differs from the more traditional null hypothesis significance testing. The results showed that the data obtained via the two online recruitment platforms, the Amazon Mechanical Turk crowdsourcing site and the virtual environment of Second Life, were statistically equivalent to the data obtained from the college sample, on the basis of means of standardized measures of psychological stress and sleep quality. Additionally, correlations between the sleep and stress measures were not statistically different between the groups. These results, along with practical considerations for the use of these recruitment platforms, are discussed, and recommendations for other researchers who may be considering the use of these platforms are provided.

Bayesian analysis in educational psychology research: An example of gender differences in achievement goals

Much research in educational psychology concerns group differences. In this study, we argue that Bayesian estimation is more appropriate for testing group differences than is the traditional null hypothesis significance testing (NHST). We demonstrate the use of Bayesian estimation on gender differences in students' achievement goals. Research findings on gender differences in achievement goals have been mixed. We explain how Bayesian estimation of mean differences is more intuitive, informative, and coherent in comparison with NHST, how it overcomes structural and interpretive problems of NHST, and how it offers a way to achieve cumulative progress toward increasing precision in estimating gender differences in achievement goals. We provide an empirical demonstration by comparing a Bayesian and a traditional NHST analysis of gender differences in achievement goals among 442 7th-grade students (223 girls and 219 boys). Whereas findings from the two analyses indicate comparable results of higher endorsement of mastery goals among girls and higher endorsement of performance-approach and avoidance goals among boys, it is the Bayesian analysis rather than the NHST that is more intuitively interpreted. We conclude by discussing the perceived disadvantages of Bayesian estimation, and some ways in which a consideration of Bayesian probability can aid interpretations of traditional analytical methods.

Extending Theory-Based Quantitative Predictions to New Health Behaviors.

Traditional null hypothesis significance testing suffers many limitations and is poorly adapted to theory testing. A proposed alternative approach, called Testing Theory-based Quantitative Predictions, uses effect size estimates and confidence intervals to directly test predictions based on theory. This paper replicates findings from previous smoking studies and extends the approach to diet and sun protection behaviors using baseline data from a Transtheoretical Model behavioral intervention (N = 5407). Effect size predictions were developed using two methods: (1) applying refined effect size estimates from previous smoking research or (2) using predictions developed by an expert panel. Thirteen of 15 predictions were confirmed for smoking. For diet, 7 of 14 predictions were confirmed using smoking predictions and 6 of 16 using expert panel predictions. For sun protection, 3 of 11 predictions were confirmed using smoking predictions and 5 of 19 using expert panel predictions. Expert panel predictions and smoking-based predictions poorly predicted effect sizes for diet and sun protection constructs. Future studies should aim to use previous empirical data to generate predictions whenever possible. The best results occur when there have been several iterations of predictions for a behavior, such as with smoking, demonstrating that expected values begin to converge on the population effect size. Overall, the study supports necessity in strengthening and revising theory with empirical data.

Measuring the crowd within again: a pre-registered replication study.

According to the crowd within effect, the average of two estimates from one person tends to be more accurate than a single estimate of that person. The effect implies that the well documented wisdom of the crowd effect—the crowd's average estimate tends to be more accurate than the individual estimates—can be obtained within a single individual. In this paper, we performed a high-powered, pre-registered replication study of the original experiment. Our replication results are evaluated with the traditional null hypothesis significance testing approach, as well as with effect sizes and their confidence intervals. We adopted a co-pilot approach, in the sense that all analyses were performed independently by two researchers using different analysis software. Moreover, we report Bayes factors for all tests. We successfully replicated the crowd within effect, both when the second guess was made immediately after the first guess, as well as when it was made 3 weeks later. The experimental protocol, the raw data, the post-processed data and the analysis code are available online.