Testing Theory-Based Quantitative Predictions with New Behaviors

Abstract

Theory development is essential for the generation and support of research ideas. Traditional Null Hypothesis Significance Testing (NHST) has been the modus operandi for testing research questions across many branches of science since the early 20th century. The focus of a statistical test under the NHST framework considers the rejection or acceptance of a null hypothesis based on a conditional probability of the data given that the null hypothesis is true (i.e. a p-value). This approach provides no direct support for a specific theory, which often takes the form of an alternative hypothesis. Furthermore, rejection of a null hypothesis based on a p-value provides no information on the magnitude of a difference and is affected by sample size, alpha level, and effect size. Such dependency on p-values can lead to misunderstanding and misinterpretation of results and conclusions. Therefore, the limitations of NHST warrant the investigation and development of new, more rigorous approaches to theory testing. A quantitative approach, called “Testing Theory-Based Quantitative Predictions” (TTQP), has been proposed using effect size indices and confidence intervals to directly test predictions posited by theory (Velicer et al, 2008). Effect size indices provide information regarding the magnitude and direction of an effect while confidence intervals provide a means of “testing” specific predictions. This approach is an iterative process, allowing the researcher to tailor the theory as empirical data is collected. The use of the TTQP approach contributes to the movement away from NHST and the reliance on p-values, while promoting a stronger and more informative method. A quantitative orientation represents an essential change in thinking about theory testing by emphasizing the numeric strength of a measure, leading researchers away from a simple binary accept/reject framework. Predictions are made relative to the specific measure/variable, but the use of effect sizes allows for comparison across studies and across theories. Therefore, the TTQP approach actually provides more information than traditional NHST. The TTQP approach involves several steps. First, verbal descriptions of the expected values are designated a priori. These predictions are theory-based and guided by previous empirical findings (e.g. “small effect”). Second, verbal predictions are translated into quantitative values based on traditional guidelines or empirical results (e.g. “0.01”). Then, observed effect size estimates with surrounding confidence intervals are generated from sample data. If a confidence interval contains the predicted value, the prediction is confirmed. If the predicted value falls outside of the confidence interval, the prediction is