Null Hypothesis Significance Testing Research Articles

Research, statistics and mathematics educators in Nigeria: effect size perspective

Over reliance on the perspective of a dichotomous reject or fail-to-reject outcome from a null hypothesis testing framework to answer research questions has become a worrisome issue to research methodologists and statistics experts. Thus, the Journals of Mathematical Association of Nigeria, Abacus (2013 & 2014) were surveyed to investigate the effect size reports practice among Mathematics Educators in Nigeria. The study showed that majority (60%) of questions of research interests of Mathematics Educators were answered by Null Hypothesis Significant Testing (NHST) and less than one –fifth (16%) of this empirical studies reported Effect sizes (standardized) for their findings. However, the survey further revealed that though Effect sizes were mostly not reported but can be conveniently estimated from the associated descriptive statistics reported by the researchers except in the analyses which seek for relationship in categorical data. Recommendations made included that Editorial Policies and guidelines of Journals in Education, especially in Mathematics Education should include Effect sizes and Confidence Intervals reports for authors.Keywords: Research, Statistics, Effect size, Mathematics Educators, Empirical studies

Null hypothesis significance testing: a short tutorial

Although thoroughly criticized, null hypothesis significance testing (NHST) remains the statistical method of choice used to provide evidence for an effect, in biological, biomedical and social sciences. In this short tutorial, I first summarize the concepts behind the method, distinguishing test of significance (Fisher) and test of acceptance (Newman-Pearson) and point to common interpretation errors regarding the p-value. I then present the related concepts of confidence intervals and again point to common interpretation errors. Finally, I discuss what should be reported in which context. The goal is to clarify concepts to avoid interpretation errors and propose reporting practices.

Measures of Difference and Significance in the Era of Computer Simulations, Meta-Analysis, and Big Data

In traditional research, repeated measurements lead to a sample of results, and inferential statistics can be used to not only estimate parameters, but also to test statistical hypotheses concerning these parameters. In many cases, the standard error of the estimates decreases (asymptotically) with the square root of the sample size, which provides a stimulus to probe large samples. In simulation models, the situation is entirely different. When probability distribution functions for model features are specified, the probability distribution function of the model output can be approached using numerical techniques, such as bootstrapping or Monte Carlo sampling. Given the computational power of most PCs today, the sample size can be increased almost without bounds. The result is that standard errors of parameters are vanishingly small, and that almost all significance tests will lead to a rejected null hypothesis. Clearly, another approach to statistical significance is needed. This paper analyzes the situation and connects the discussion to other domains in which the null hypothesis significance test (NHST) paradigm is challenged. In particular, the notions of effect size and Cohen’s d provide promising alternatives for the establishment of a new indicator of statistical significance. This indicator attempts to cover significance (precision) and effect size (relevance) in one measure. Although in the end more fundamental changes are called for, our approach has the attractiveness of requiring only a minimal change to the practice of statistics. The analysis is not only relevant for artificial samples, but also for present-day huge samples, associated with the availability of big data.

Using the Coefficient of Confidence to Make the Philosophical Switch From A Posteriori to A Priori Inferential Statistics.

There has been much controversy over the null hypothesis significance testing procedure, with much of the criticism centered on the problem of inverse inference. Specifically, p gives the probability of the finding (or one more extreme) given the null hypothesis, whereas the null hypothesis significance testing procedure involves drawing a conclusion about the null hypothesis given the finding. Many critics have called for null hypothesis significance tests to be replaced with confidence intervals. However, confidence intervals also suffer from a version of the inverse inference problem. The only known solution to the inverse inference problem is to use the famous theorem by Bayes, but this involves commitments that many researchers are not willing to make. However, it is possible to ask a useful question for which inverse inference is not a problem and that leads to the computation of the coefficient of confidence. In turn, and much more important, using the coefficient of confidence implies the desirability of switching from the current emphasis on a posteriori inferential statistics to an emphasis on a priori inferential statistics.

Hypothesis Testing in the Real World.

Critics of null hypothesis significance testing suggest that (a) its basic logic is invalid and (b) it addresses a question that is of no interest. In contrast to (a), I argue that the underlying logic of hypothesis testing is actually extremely straightforward and compelling. To substantiate that, I present examples showing that hypothesis testing logic is routinely used in everyday life. These same examples also refute (b) by showing circumstances in which the logic of hypothesis testing addresses a question of prime interest. Null hypothesis significance testing may sometimes be misunderstood or misapplied, but these problems should be addressed by improved education.

Observation-Oriented Modeling: Going Beyond "Is It All a Matter of Chance"?

An alternative to null hypothesis significance testing is presented and discussed. This approach, referred to as observation-oriented modeling, is centered on model building in an effort to explicate the structures and processes believed to generate a set of observations. In terms of analysis, this novel approach complements traditional methods based on means, variances, and covariances with methods of pattern detection and analysis. Using data from a previously published study by Shoda et al., the basic tenets and methods of observation-oriented modeling are demonstrated and compared with traditional methods, particularly with regard to null hypothesis significance testing.

Thou Shalt Not Bear False Witness Against Null Hypothesis Significance Testing.

Null hypothesis significance testing (NHST) has been the subject of debate for decades and alternative approaches to data analysis have been proposed. This article addresses this debate from the perspective of scientific inquiry and inference. Inference is an inverse problem and application of statistical methods cannot reveal whether effects exist or whether they are empirically meaningful. Hence, raising conclusions from the outcomes of statistical analyses is subject to limitations. NHST has been criticized for its misuse and the misconstruction of its outcomes, also stressing its inability to meet expectations that it was never designed to fulfil. Ironically, alternatives to NHST are identical in these respects, something that has been overlooked in their presentation. Three of those alternatives are discussed here (estimation via confidence intervals and effect sizes, quantification of evidence via Bayes factors, and mere reporting of descriptive statistics). None of them offers a solution to the problems that NHST is purported to have, all of them are susceptible to misuse and misinterpretation, and some bring around their own problems (e.g., Bayes factors have a one-to-one correspondence with p values, but they are entirely deprived of an inferential framework). Those alternatives also fail to cover a broad area of inference not involving distributional parameters, where NHST procedures remain the only (and suitable) option. Like knives or axes, NHST is not inherently evil; only misuse and misinterpretation of its outcomes needs to be eradicated.

Tests of Statistical Significance Made Sound.

This article considers the nature and place of tests of statistical significance (ToSS) in science, with particular reference to psychology. Despite the enormous amount of attention given to this topic, psychology's understanding of ToSS remains deficient. The major problem stems from a widespread and uncritical acceptance of null hypothesis significance testing (NHST), which is an indefensible amalgam of ideas adapted from Fisher's thinking on the subject and from Neyman and Pearson's alternative account. To correct for the deficiencies of the hybrid, it is suggested that psychology avail itself of two important and more recent viewpoints on ToSS, namely the neo-Fisherian and the error-statistical perspectives. The neo-Fisherian perspective endeavors to improve on Fisher's original account and rejects key elements of Neyman and Pearson's alternative. In contrast, the error-statistical perspective builds on the strengths of both statistical traditions. It is suggested that these more recent outlooks on ToSS are a definite improvement on NHST, especially the error-statistical position. It is suggested that ToSS can play a useful, if limited, role in psychological research. At the end, some lessons learnt from the extensive debates about ToSS are presented.

The Need for Nuance in the Null Hypothesis Significance Testing Debate.

Null hypothesis significance testing (NHST) provides an important statistical toolbox, but there are a number of ways in which it is often abused and misinterpreted, with bad consequences for the reliability and progress of science. Parts of contemporary NHST debate, especially in the psychological sciences, is reviewed, and a suggestion is made that a new distinction between strongly, weakly, and very weakly anti-NHST positions is likely to bring added clarity to the debate.

Investigating the Effect of Gaze Cues and Emotional Expressions on the Affective Evaluations of Unfamiliar Faces.

People look at what they are interested in, and their emotional expressions tend to indicate how they feel about the objects at which they look. The combination of gaze direction and emotional expression can therefore convey important information about people’s evaluations of the objects in their environment, and can even influence the subsequent evaluations of those objects by a third party, a phenomenon known as the emotional gaze effect. The present study extended research into the effect of emotional gaze cues by investigating whether they affect evaluations of the most important aspect of our social environment–other people–and whether the presence of multiple gaze cues enhances this effect. Over four experiments, a factorial within-subjects design employing both null hypothesis significance testing and a Bayesian statistical analysis replicated previous work showing an emotional gaze effect for objects, but found strong evidence that emotional gaze cues do not affect evaluations of other people, and that multiple, simultaneously presented gaze cues do not enhance the emotional gaze effect for either the evaluations of objects or of people. Overall, our results suggest that emotional gaze cues have a relatively weak influence on affective evaluations, especially of those aspects of our environment that automatically elicit affectively valenced reactions, including other humans.

Searching for Significance in the Scholarship of Teaching and Learning and Finding None: Understanding Non-Significant Results

Quantitative results from empirical studies are common in the field of Scholarship of Teaching and Learning (SoTL), but it is important to remain aware of what the results from our studies can, and cannot, tell us. Oftentimes studies conducted to examine teaching and learning are constrained by class size. Small sample sizes negatively influence statistical power and make non-significant results a more likely occurrence. When one finds non-significant results it is important to consider what conclusions can be drawn from the study. This article provides information on null hypothesis significance testing that is relevant to our understanding of non-significant results, and it highlights the importance of recognizing underpowered studies in the teaching and learning literature. Factors that can contribute to non-significant findings in a study are also highlighted. Being aware of these factors, statistical power, and the logic of significance testing will put scholars in a better position to evaluate non-significant results from their own research and that of others.

A Tutorial on Hunting Statistical Significance by Chasing N.

There is increasing concern about the replicability of studies in psychology and cognitive neuroscience. Hidden data dredging (also called p-hacking) is a major contributor to this crisis because it substantially increases Type I error resulting in a much larger proportion of false positive findings than the usually expected 5%. In order to build better intuition to avoid, detect and criticize some typical problems, here I systematically illustrate the large impact of some easy to implement and so, perhaps frequent data dredging techniques on boosting false positive findings. I illustrate several forms of two special cases of data dredging. First, researchers may violate the data collection stopping rules of null hypothesis significance testing by repeatedly checking for statistical significance with various numbers of participants. Second, researchers may group participants post hoc along potential but unplanned independent grouping variables. The first approach ‘hacks’ the number of participants in studies, the second approach ‘hacks’ the number of variables in the analysis. I demonstrate the high amount of false positive findings generated by these techniques with data from true null distributions. I also illustrate that it is extremely easy to introduce strong bias into data by very mild selection and re-testing. Similar, usually undocumented data dredging steps can easily lead to having 20–50%, or more false positives.

Effects of Velocity Loss During Resistance Training on Performance in Professional Soccer Players

To analyze the effects of 2 resistance-training (RT) programs that used the same relative loading but different repetition volume, using the velocity loss during the set as the independent variable: 15% (VL15) vs 30% (VL30). Sixteen professional soccer players with RT experience (age 23.8 ± 3.5 y, body mass 75.5 ± 8.6 kg) were randomly assigned to 2 groups, VL15 (n = 8) or VL30 (n = 8), that followed a 6-wk (18-session) velocity-based squat-training program. Repetition velocity was monitored in all sessions. Assessments performed before (Pre) and after training (Post) included estimated 1-repetition maximum (1RM) and change in average mean propulsive velocity (AMPV) against absolute loads common to Pre and Post tests, countermovement jump (CMJ), 30-m sprint (T30), and Yo-Yo Intermittent Recovery Test (YIRT). Null-hypothesis significance testing and magnitude-based-inference statistical analyses were performed. VL15 obtained greater gains in CMJ height than VL30 (P < .05), with no significant differences between groups for the remaining variables. VL15 showed a likely/possibly positive effect on 1RM (91/9/0%), AMPV (73/25/2%), and CMJ (87/12/1%), whereas VL30 showed possibly/unclear positive effects on 1RM (65/33/2%) and AMPV (46/36/18%) and possibly negative effects on CMJ (4/38/57%). The effects on T30 performance were unclear/unlikely for both groups, whereas both groups showed most likely/likely positive effects on YIRT. A velocity-based RT program characterized by a low degree of fatigue (15% velocity loss in each set) is effective to induce improvements in neuromuscular performance in professional soccer players with previous RT experience.

DAPAR & ProStaR: software to perform statistical analyses in quantitative discovery proteomics.

SummaryDAPAR and ProStaR are software tools to perform the statistical analysis of label-free XIC-based quantitative discovery proteomics experiments. DAPAR contains procedures to filter, normalize, impute missing value, aggregate peptide intensities, perform null hypothesis significance tests and select the most likely differentially abundant proteins with a corresponding false discovery rate. ProStaR is a graphical user interface that allows friendly access to the DAPAR functionalities through a web browser.Availability and implementationDAPAR and ProStaR are implemented in the R language and are available on the website of the Bioconductor project (http://www.bioconductor.org/). A complete tutorial and a toy dataset are accompanying the packages.

Bayesian Dependence Tests for Continuous, Binary and Mixed Continuous-Binary Variables

Tests for dependence of continuous, discrete and mixed continuous-discrete variables are ubiquitous in science. The goal of this paper is to derive Bayesian alternatives to frequentist null hypothesis significance tests for dependence. In particular, we will present three Bayesian tests for dependence of binary, continuous and mixed variables. These tests are nonparametric and based on the Dirichlet Process, which allows us to use the same prior model for all of them. Therefore, the tests are “consistent” among each other, in the sense that the probabilities that variables are dependent computed with these tests are commensurable across the different types of variables being tested. By means of simulations with artificial data, we show the effectiveness of the new tests.

A common misapplication of statistical inference: Nuisance control with null-hypothesis significance tests

Experimental research on behavior and cognition frequently rests on stimulus or subject selection where not all characteristics can be fully controlled, even when attempting strict matching. For example, when contrasting patients to controls, variables such as intelligence or socioeconomic status are often correlated with patient status. Similarly, when presenting word stimuli, variables such as word frequency are often correlated with primary variables of interest. One procedure very commonly employed to control for such nuisance effects is conducting inferential tests on confounding stimulus or subject characteristics. For example, if word length is not significantly different for two stimulus sets, they are considered as matched for word length. Such a test has high error rates and is conceptually misguided. It reflects a common misunderstanding of statistical tests: interpreting significance not to refer to inference about a particular population parameter, but about 1. the sample in question, 2. the practical relevance of a sample difference (so that a nonsignificant test is taken to indicate evidence for the absence of relevant differences). We show inferential testing for assessing nuisance effects to be inappropriate both pragmatically and philosophically, present a survey showing its high prevalence, and briefly discuss an alternative in the form of regression including nuisance variables.

Researchers' choice of the number and range of levels in experiments affects the resultant variance-accounted-for effect size.

In psychology, the reporting of variance-accounted-for effect size indices has been recommended and widely accepted through the movement away from null hypothesis significance testing. However, most researchers have paid insufficient attention to the fact that effect sizes depend on the choice of the number of levels and their ranges in experiments. Moreover, the functional form of how and how much this choice affects the resultant effect size has not thus far been studied. We show that the relationship between the population effect size and number and range of levels is given as an explicit function under reasonable assumptions. Counterintuitively, it is found that researchers may affect the resultant effect size to be either double or half simply by suitably choosing the number of levels and their ranges. Through a simulation study, we confirm that this relation also applies to sample effect size indices in much the same way. Therefore, the variance-accounted-for effect size would be substantially affected by the basic research design such as the number of levels. Simple cross-study comparisons and a meta-analysis of variance-accounted-for effect sizes would generally be irrational unless differences in research designs are explicitly considered.

Performing Inferential Statistics Prior to Data Collection.

Typically, in education and psychology research, the investigator collects data and subsequently performs descriptive and inferential statistics. For example, a researcher might compute group means and use the null hypothesis significance testing procedure to draw conclusions about the populations from which the groups were drawn. We propose an alternative inferential statistical procedure that is performed prior to data collection rather than afterwards. To use this procedure, the researcher specifies how close she or he desires the group means to be to their corresponding population means and how confident she or he wishes to be that this actually is so. We derive an equation that provides researchers with a way to determine the sample size needed to meet the specifications concerning closeness and confidence, regardless of the number of groups.

Null hypothesis significance testing: a short tutorial

Although thoroughly criticized, null hypothesis significance testing (NHST) remains the statistical method of choice used to provide evidence for an effect, in biological, biomedical and social sciences. In this short tutorial, I first summarize the concepts behind the method, distinguishing test of significance (Fisher) and test of acceptance (Newman-Pearson) and point to common interpretation errors regarding the p-value. I then present the related concepts of confidence intervals and again point to common interpretation errors. Finally, I discuss what should be reported in which context. The goal is to clarify concepts to avoid interpretation errors and propose reporting practices.

Null hypothesis significance testing: a short tutorial

Although thoroughly criticized, null hypothesis significance testing (NHST) remains the statistical method of choice used to provide evidence for an effect, in biological, biomedical and social sciences. In this short tutorial, I first summarize the concepts behind the method, distinguishing test of significance (Fisher) and test of acceptance (Newman-Pearson) and point to common interpretation errors regarding the p-value. I then present the related concepts of confidence intervals and again point to common interpretation errors. Finally, I discuss what should be reported in which context. The goal is to clarify concepts to avoid interpretation errors and propose reporting practices.