Null Hypothesis Statistical Testing Research Articles

The psychological reality of the learned “p < .05” boundary

The .05 boundary within Null Hypothesis Statistical Testing (NHST) “has made a lot of people very angry and been widely regarded as a bad move” (to quote Douglas Adams). Here, we move past meta-scientific arguments and ask an empirical question: What is the psychological standing of the .05 boundary for statistical significance? We find that graduate students in the psychological sciences show a boundary effect when relating p-values across .05. We propose this psychological boundary is learned through statistical training in NHST and reading a scientific literature replete with “statistical significance”. Consistent with this proposal, undergraduates do not show the same sensitivity to the .05 boundary. Additionally, the size of a graduate student’s boundary effect is not associated with their explicit endorsement of questionable research practices. These findings suggest that training creates distortions in initial processing of p-values, but these might be dampened through scientific processes operating over longer timescales.

Addressing Limitations of p in Counseling Research through Reporting of Bayes Factor Bound and Effect Size Precision

Null hypothesis statistical testing (NHST) in its present form continues to be the predominant method of quantitative research. Bayesian methods, which emphasize the probability of a hypothesis given the data, may be growing in popularity due to the advent of statistical software employing these methods. Rather than discarding the present methods of NHST, the purpose of this article is to enhance understanding of NHST for counseling researchers and provide an augmentation to presenting results that address the known limitations of counseling research and other social science research. Such recommendations include summarizing previous recommendations of employing visuals to explain data, incorporating effect size confidence intervals, and utilizing the Bayes factor bound to supplement p-values and explain the probabilities related to acceptance or rejection of the null hypothesis.

Evaluating the Impact of Supplemental Computer-Assisted Math Instruction in Elementary School: A Conceptual Replication

Despite the surge in efficacy studies, the relative effectiveness of any two math interventions is largely unknown and replications are rare. To ensure that students benefit from high-quality learning experiences, the current conceptual replication of Wang and Woodworth, a randomized control trial, evaluated the relative impacts of two computer programs used in a school district as supplements to students’ regular education math instruction, DreamBox Learning and Zearn Math. For 115 students in kindergarten and first grade, data were collected before and after an 11-week implementation period. Measures of effect size, but not null hypothesis statistical tests, suggested that participants in the DreamBox condition had higher test scores on posttest numeracy and geometry (Hedges g = .32 and .14). There was also a statistically significant group by pretest interaction, evidence of “Matthew Effects” for numeracy. Although some support for adopting DreamBox as a supplemental method for improving math competencies of students in early elementary school was provided, more research is needed as the true effect may be smaller than what the present study reports. The study’s implications are discussed.

Beyond Statistical Significance: A Holistic View of What Makes a Research Finding "Important"

Students often believe that statistical significance is the only determinant of whether a quantitative result is “important.” In this paper, I review traditional null hypothesis statistical testing to identify what questions inferential statistics can and cannot answer, including statistical significance, effect size and direction, causality, generalizability, and changeability of the independent variable. I illustrate these issues with examples from an empirical study of the association between how much time teenagers spent playing video games and time spent reading. I describe how study design and context determine each of those aspects of “importance,” and close by summarizing how to provide a holistic view of importance when writing about a quantitative analysis. I also include exercises to guide students through applying these concepts to articles in newspapers and scholarly journals.

Bayesian Performance Analysis for Algorithm Ranking Comparison

In the field of optimization and machine learning, the statistical assessment of results has played a key role in conducting algorithmic performance comparisons. Classically, null hypothesis statistical tests have been used. However, recently, alternatives based on Bayesian statistics have shown great potential in complex scenarios, especially when quantifying the uncertainty in the comparison. In this work, we delve deep into the Bayesian statistical assessment of experimental results by proposing a framework for the analysis of several algorithms on several problems/instances. To this end, experimental results are transformed to their corresponding rankings of algorithms, assuming that these rankings have been generated by a probability distribution (defined on permutation spaces). From the set of rankings, we estimate the posterior distribution of the parameters of the studied probability models, and several inferences concerning the analysis of the results are examined. Particularly, we study questions related to the probability of having one algorithm in the first position of the ranking or the probability that two algorithms are in the same relative position in the ranking. Not limited to that, the assumptions, strengths, and weaknesses of the models in each case are studied. To help other researchers to make use of this kind of analysis, we provide a Python package and source code implementation at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://zenodo.org/record/6320599</uri> .

Remarkably reproducible psychological (memory) phenomena in the classroom: some evidence for generality from small-N research

BackgroundMainstream psychology is experiencing a crisis of confidence. Many of the methodological solutions offered in response have focused largely on statistical alternatives to null hypothesis statistical testing, ignoring nonstatistical remedies that are readily available within psychology; namely, use of small-N designs. In fact, many classic memory studies that have passed the test of replicability used them. That methodological legacy warranted a retrospective look at nonexperimental data to explore the generality of the reported effects.MethodVarious classroom demonstrations were conducted over multiple semesters in introductory psychology courses with typical, mostly freshman students from a predominantly white private Catholic university in the US Midwest based on classic memory experiments on immediate memory span, chunking, and depth of processing.ResultsStudents tended to remember 7 ± 2 digits, remembered more digits of π following an attached meaningful story, and remembered more words after elaborative rehearsal than after maintenance rehearsal. These results amount to replications under uncontrolled classroom environments of the classic experiments originally conducted largely outside of null hypothesis statistical testing frameworks.ConclusionsIn light of the ongoing replication crisis in psychology, the results are remarkable and noteworthy, validating these historically important psychological findings. They are testament to the reliability of reproducible effects as the hallmark of empirical findings in science and suggest an alternative approach to commonly proffered solutions to the replication crisis.

ORMA: A strategy to reduce Psychology's replication problems

Because of their historical reliance upon null hypothesis statistical tests (NHST), the human sciences have developed a number of potentially problematic research literatures. While aware of the file drawer effect since the 1970s, scientists have been largely unsuccessful at addressing its pernicious effects. Because significant results have a greater likelihood of being published than do nonsignificant effects, many of our research literatures might currently be constructed upon a series of Type I errors and inflated effect sizes. A method (called Original Replication of Meta-Analyses or ORMA) has recently been developed for identifying problematic research literatures and offering a method to address the problems due to publication bias. Philosophers of science have long argued that a chief reason for science's preeminence as a source of knowledge rests in its ability to self-correct. Researchers in the human sciences are now able to empirically test their research literatures to ascertain which are in need of repair. The use of ORMA serves to lessen the problems that led to the recent calls for bans on significant/nonsignificant statistics in human science research. ORMA will also improve psychology's ability to successfully replicate its research findings.

Beyond P-Value-Obsession: When are Statistical Hypothesis Tests Required and Appropriate?

Complementing more specific “p-value discussions”, this paper presents fundamental arguments for when null hypothesis statistical significance tests (NHST) are required and appropriate. The arguments, which are paradigmatic rather than technical, are operationalised and broken down to the extent that their logic can be mapped into a decision tree for the use of NHST. We derive a perspective that does not ban p-values but proposes to minimize their use. P-values will become rather rare in (agricultural) economics if they are not applied in any cases, where the conditions for their proper use are violated or where their use is not appropriate or required in order to answer the questions asked of the data. The accompanying shift from prioritising inferential statistics to recognising the value of descriptive statistics requires not only a change in entrenched habits of thought. This shift also has the potential to trigger changes in the research processes and in the evaluation of new approaches within the disciplines.

In search of a better reporting of scientific results: A data probability language

A recent paper published in Trends in Ecology and Evolution suggested a new alternative for the reporting of statistical results, using a language based on evidence against the null hypothesis. I agree that the reporting of null hypothesis statistical testing clearly needs improvement, but the proposal of an evidence-based language has several drawbacks: a) it goes back to the original Fisherian continuous interpretation of p-values, b) at the same time uses some loose categorizations and, c) most importantly, it may provide a wrong idea of what p-values actually are. By saying that there is very strong, strong, moderate, weak or little evidence of an effect, the reader gets the idea that p-values are providing Bayesian-type information on the probability of the null hypothesis given our data. However, p-values are only providing information on the probability of having obtained our data (or more extreme data), under the trueness of the null hypothesis. That is why I suggest reporting results using a data probability-based language, together with a previous and separate specification of the magnitude of the effects.

Comparing Two Samples Through Stochastic Dominance: A Graphical Approach

Nondeterministic measurements are common in real-world scenarios: the performance of a stochastic optimization algorithm or the total reward of a reinforcement learning agent in a chaotic environment are just two examples in which unpredictable outcomes are common. These measures can be modeled as random variables and compared among each other via their expected values or more sophisticated tools such as null hypothesis statistical tests. In this article, we propose an alternative framework to visually compare two samples according to their estimated cumulative distribution functions. First, we introduce a dominance measure for two random variables that quantifies the proportion in which the cumulative distribution function of one of the random variables stochastically dominates the other one. Then, we present a graphical method that decomposes in quantiles (i) the proposed dominance measure and (ii) the probability that one of the random variables takes lower values than the other. With illustrative purposes, we reevaluate the experimentation of an already published work with the proposed methodology and we show that additional conclusions—missed by the rest of the methods—can be inferred. Additionally, the software package RVCompare was created as a convenient way of applying and experimenting with the proposed framework.

Think like a Bayesian and avoid pitfalls from our frequentist past

Bayesian inference is a powerful tool that is increasingly being used by ecologists. This is largely due to the flexibility in model specification and improvements in software that makes this tool easier to use. However, with increasing ease of use comes a risk of misuse or abuse. We review four major issues we have identified in the use of Bayesian methods and offer reminders and suggestions that will improve the application and reporting of Bayesian inference while at the same time, hopefully, avoiding the pitfalls that have plagued null hypothesis statistical testing (NHST). These issues include; 1) understanding software and model specification; 2) use of prior probability distributions; 3) maximizing utility of posterior probability distributions; and 4) avoiding dichotomous thinking (i.e., the NHST pitfall). We suggest ecologists should strive for openness in their use of statistical software by understanding their model and providing the full computer code used, develop reasonable and informative priors, and make full use of posterior information that Bayesian methods provide. At the same time, ecologists should avoid dichotomizing results into significant/ non-significant boxes, eliminate null hypothesis tests (including probability intervals for hypothesis testing), and use clear language when describing results. Finally, quantitative training should be expanded in undergraduate curricula to provide students with a larger suite of foundational core concepts that extend beyond NHST.

How do social factors relate to blunted facial affect in schizophrenia? A digital phenotyping study using ambulatory video recordings

Clinical interviews and laboratory-based emotional induction paradigms provide consistent evidence that facial affect is blunted in many individuals with schizophrenia. Although it is clear that blunted facial affect is not a by-product of diminished emotional experience in schizophrenia, factors contributing to blunted affect remain unclear. The current study used a combination of ambulatory video recordings that were evaluated via computerized facial affect analysis and concurrently completed ecological momentary assessment surveys to assess whether blunted affect reflects insufficient reactivity to affective or contextual factors. Specifically, whether individuals with schizophrenia require more intense affective experiences to produce expression, or whether they are less reactive to social factors (i.e. being in the presence of others, social motivation). Participants included outpatients with schizophrenia (n = 33) and healthy controls (n = 31) who completed six days of study procedures. Multilevel linear models were evaluated using both Null-Hypothesis Statistical Testing and Bayesian analyses. Individuals with schizophrenia displayed comparable expression of positive and negative emotion to controls during daily life, and no evidence was found for a different intensity of experience required for expression in either group. However, social factors differentially influenced facial expression in schizophrenia compared to controls, such that individuals with schizophrenia did not modulate their expressions based on social motivation to the same extent as controls. These findings suggest that social motivation may play an important role in determining when blunting occurs.

Hypothesis Statement and Statistical Testing: A Tutorial

Many researchers and beginners in social research have several dilemmas and confusion in their mind about hypothesis statement and statistical testing of hypotheses. A distinction between the research hypothesis and statistical hypotheses, and understanding the limitations of the historically used null hypothesis statistical testing, is useful in clarifying these doubts. This article presents some data from the published research articles to support the view that the is format as well as the will format is appropriate to stating hypotheses. The article presents a social research framework to present the research hypothesis and statistical hypotheses is proper perspective.

Reverse Bayesian Implications of p-Values Reported in Critical Care Randomized Trials.

BackgroundMisinterpretations of the p-value in null-hypothesis statistical testing are common. We aimed to determine the implications of observed p-values in critical care randomized controlled trials (RCTs).MethodsWe included three cohorts of published RCTs: Adult-RCTs reporting a mortality outcome, Pediatric-RCTs reporting a mortality outcome, and recent Consecutive-RCTs reporting p-value ≤.10 in six higher-impact journals. We recorded descriptive information from RCTs. Reverse Bayesian implications of obtained p-values were calculated, reported as percentages with inter-quartile ranges.ResultsObtained p-value was ≤.005 in 11/216 (5.1%) Adult-RCTs, 2/120 (1.7%) Pediatric-RCTs, and 37/90 (41.1%) Consecutive-RCTs. An obtained p-value .05–.0051 had high False Positive Rates; in Adult-RCTs, minimum (assuming prior probability of the alternative hypothesis was 50%) and realistic (assuming prior probability of the alternative hypothesis was 10%) False Positive Rates were 16.7% [11.2, 21.8] and 64.3% [53.2, 71.4]. An obtained p-value ≤.005 had lower False Positive Rates; in Adult-RCTs the realistic False Positive Rate was 7.7% [7.7, 16.0]. The realistic probability of the alternative hypothesis for obtained p-value .05–.0051 (ie, Positive Predictive Value) was 28.0% [24.1, 34.8], 30.6% [27.7, 48.5], 29.3% [24.3, 41.0], and 32.7% [24.1, 43.5] for Adult-RCTs, Pediatric-RCTs, Consecutive-RCTs primary and secondary outcome, respectively. The maximum Positive Predictive Value for p-value category .05–.0051 was median 77.8%, 79.8%, 78.8%, and 81.4% respectively. To have maximum or realistic Positive Predictive Value >90% or >80%, RCTs needed to have obtained p-value ≤.005. The credibility of p-value .05–.0051 findings were easy to challenge, and the credibility to rule-out an effect with p-value >.05 to .10 was low. The probability that a replication study would obtain p-value ≤.05 did not approach 90% unless the obtained p-value was ≤.005.ConclusionsUnless the obtained p-value was ≤.005, the False Positive Rate was high, and the Positive Predictive Value and probability of replication of “statistically significant” findings were low.

Bayesian inference of population prevalence

Scientists use statistical tools to evaluate observations or measurements from carefully designed experiments. In psychology and neuroscience, these experiments involve studying a randomly selected group of people, looking for patterns in their behaviour or brain activity, to infer things about the population at large. The usual method for evaluating the results of these experiments is to carry out null hypothesis statistical testing (NHST) on the population mean – that is, the average effect in the population that the study participants were selected from. The test asks whether the observed results in the group studied differ from what might be expected if the average effect in the population was zero. However, in psychology and neuroscience studies, people’s brain activity and performance on cognitive tasks can differ a lot. This means important effects in individuals can be lost in the overall population average. Ince et al. propose that this shortcoming of NHST can be overcome by shifting the statistical analysis away from the population mean, and instead focusing on effects in individual participants. This led them to create a new statistical approach named Bayesian prevalence. The method looks at effects within each individual in the study and asks how likely it would be to see the same result if the experiment was repeated with a new person chosen from the wider population at random. Using this approach, it is possible to quantify how typical or uncommon an observed effect is in the population, and the uncertainty around this estimate. This differs from NHST which only provides a binary ‘yes or no’ answer to the question, ‘does this experiment provide sufficient evidence that the average effect in the population is not zero?’ Another benefit of Bayesian prevalence is that it can be applied to studies with small numbers of participants which cannot be analysed using other statistical methods. Ince et al. show that the Bayesian prevalence can be applied to a range of psychology and neuroimaging experiments, from brain imaging to electrophysiology studies. Using this alternative statistical method could help address issues of replication in these fields where NHST results are sometimes not the same when studies are repeated.

Why We All Should Be Bayesians: An Introduction to Bayesian Analysis

The purpose of this symposium is to introduce management researchers to the opportunities of Bayesian statistics for empirical research in the management sciences. We will outline the fundamental features of the Bayesian method without delving into the mathematical details. Instead, we will outline the conceptual differences and potential advantages of a Bayesian approach compared to traditional statistical analyses involving null-hypothesis statistical significance tests. We will then introduce an illustrative example from empirical management research that illustrates opportunities for the useful application of Bayesian data analysis and introduces participants on how to interpret the outcomes of Bayesian analyses for causal inference. Finally, we will discuss why in spite of strong arguments supporting the use of Bayesian statistics, the field of management research has been very reluctant considering Bayesian analysis as an alternative and how management scholars can support related methodological change efforts. The purpose of this symposium is to convince participants of the potential opportunities Bayesian methods can provide and to motivate organizational researchers to consider Bayesian statistics as an alternative in future research.

Contemporary Issues in Reporting Statistical, Practical, and Clinical Significance in Counseling Research

Null hypothesis statistical testing (NHST) represents a primary method of addressing quantitative results in counseling research. The use of NHST and the expression of results are often limited to the populations of interest, and many studies may not be replicable. Furthermore, the communication of results in terms of practical and clinical significance is not simply an extension of NHST but rather a necessary endeavor to communicate accurate and useful results. Communication and elucidation of effect size and confidence intervals, presentation of visual depictions of data, and alternative measures to express clinical significance are highlighted. Limitations of NHST, the necessary results to illuminate findings related to NHST, and heuristic examples for counseling researchers when reporting the results of their studies are provided. Further emphasis on the aforementioned areas will enhance understanding of results to stakeholders and increase the functionality and scholarly utility of counseling research.

An investigation of the effects of chaotic maps on the performance of metaheuristics

AbstractThis article presents an empirical investigation of the effects of chaotic maps on the performance of metaheuristics. Particle Swarm Optimization and Simulated Annealing are modified to use chaotic maps instead of the traditional pseudorandom number generators and then compared on five common benchmark functions using nonparametric null hypothesis statistical testing. Contrary to what has often been assumed, results show that chaotic maps do not generally appear to increase the performance of swarm metaheuristics in a statistically significant way, except possibly for noisy functions. No performance differences were observed with the single‐state Simulated Annealing algorithm. Finally, it is shown that sequence effects may be responsible for the observed performance increase. These findings reveal new research directions in using chaotic maps for metaheuristics research. The MATLAB code used in this article is available in a GitHub repository for suggestions and/or corrections.

Expanding the toxicologist's statistical toolbox: Using effect size estimation and dose-response modelling for holistic assessments instead of generic testing

It is tempting to base (eco-)toxicological assay evaluation solely on statistical significance tests. The approach is stringent, objective and facilitates binary decisions. However, tests according to null hypothesis statistical testing (NHST) are thought experiments that rely heavily on assumptions. The generic and unreflected application of statistical tests has been called “mindless” by Gigerenzer. While statistical tests have an appropriate application domain, the present work investigates how unreflected testing may affect toxicological assessments. Dunnett multiple-comparison and Williams trend testing and their compatibility intervals are compared with dose-response-modelling in case studies, where data do not follow textbook behavior, nor behave as expected from a toxicological point of view. In such cases, toxicological assessments based only on p-values may be biased and biological evaluations based on plausibility may be prioritized. If confidence in a negative assay outcome cannot be established, further data may be needed for a robust toxicological assessment.

Historically recontextualizing Sidman's Tactics: How behavior analysis avoided psychology's methodological Ouroboros.

Psychology is undergoing major cultural changes methodologically, with efforts to redefine how psychologists analyze and report their data. Davidson (2018) argued that psychology's methodological crises stem from mechanical objectivity involving the adoption of an analytic tool as source of dependable knowledge. This has led to institutionalization, and eventually uncritical ritualistic use, such as happened with null hypothesis statistical testing. Davidson invoked the mythological symbol of the Ouroboros to represent the endless churning of statistical fads. Sidman (1960), in his Tactics of Scientific Research provided a shield from these problems in terms of the premium he placed on the experience, expertise, judgement, and decision-making of the scientist, that appear to be absent in psychology's ritualized processes.