Use Of Effect Sizes Research Articles

When effect sizes disagree: The case of r and d.

The increased use of effect sizes in single studies and meta-analyses raises new questions about statistical inference. Choice of an effect-size index can have a substantial impact on the interpretation of findings. The authors demonstrate the issue by focusing on two popular effect-size measures, the correlation coefficient and the standardized mean difference (e.g., Cohen's d or Hedges's g), both of which can be used when one variable is dichotomous and the other is quantitative. Although the indices are often practically interchangeable, differences in sensitivity to the base rate or variance of the dichotomous variable can alter conclusions about the magnitude of an effect depending on which statistic is used. Because neither statistic is universally superior, researchers should explicitly consider the importance of base rates to formulate correct inferences and justify the selection of a primary effect-size statistic.

Robust testing of level changes in interrupted time-series analysis

Ramsey and Ramsey [Ramsey, P.P. and Ramsey, P.H., 2003, Comparing lease-squares lines for testing level changes in interrupted time-series analysis. Journal of Statistical Computation and Simulation, 73, 31–44.] have shown that a composite procedure (CP) can provide accurate tests of level changes in interrupted time-series analysis. A CP combines the Mann–Wald procedure [Gottman, J.M., 1981, Time-Series Analysis (New York: Cambridge University Press): Mann, H.B. and Wald, A., 1943, On the statistical treatment of linear stochastic difference equations. Econometrica, 11, 173–220.] with the Kromrey and Foster-Johnson [Kromrey, J.D. and Foster-Johnson, L., 1996, Determining the efficacy of intervention: the use of effect sizes for data analysis in single-subject research. The Journal of Experimental Education, 65, 73–93.] regression procedure allowing for testing changes in level, as well as changes in both trend and level in the presence of trend effects in time-series data. Monte Carlo simulation was used to demonstrate Type I error rates for sample sizes from 5 to 20. A variety of composite procedures were found to provide such control.

The use of effect size indices to determine practical significance

The determination of significance of differences in means and of relationships between variables is of importance in many empirical studies. Usually only statistical significance is reported, which does not necessarily indicate an important (practically significant) difference or relationship. With studies based on probability samples, effect size indices should be reported in addition to statistical significance tests in order to comment on practical significance. Where complete populations or convenience samples are worked with, the determination of statistical significance is strictly speaking no longer relevant, while the effect size indices can be used as a basis to judge significance. In this article attention is paid to the use of effect size indices in order to establish practical significance. It is also shown how these indices are utilized in a few fields of statistical application and how it receives attention in statistical literature and computer packages. The use of effect sizes is illustrated by a few examples from the research literature.

Use of effect size to determine optimal dose of duloxetine in major depressive disorder

Objective At effective doses, patients with major depressive disorder (MDD) treated with duloxetine have been found to experience significant symptom improvement as measured by HAMD 17 total score. In addition, duloxetine-treated patients have significantly higher remission and response rates compared with placebo. The objective of this analysis is to determine the optimal dose of duloxetine in MDD. Materials and methods Effect size for duloxetine 40 mg, 60 mg, 80 mg, and 120 mg per day were estimated using all 6 acute phase III clinical trials in patients with MDD. The tolerability of duloxetine 40 mg, 60 mg, 80 mg, and 120 mg were evaluated using pooled data from the 6 studies. The primary efficacy measure in all trials was the HAMD 17 total score, from which were determined the effect size for HAMD 17 change scores, response rates (50% reduction from baseline to endpoint), and remission rates (HAMD 17 total score ⩽7). Results A total of 1619 randomized patients were included in these studies, of which 632 were treated with placebo; 177 with duloxetine 40 mg/day; 251 with 60 mg/day; 363 with 80 mg/day; and 196 with 120 mg/day. An evaluation of increments in effect size between doses consistently showed that the most notable gain in effect size for efficacy was the 40–60 mg/day dosage range. All dosages from 60 to 120 mg were effective. The tolerability assessment indicated duloxetine at 40–120 mg/day is well tolerated. Furthermore, the initial doses of 40–80 mg/day were found to have comparable tolerability. Conclusions The effect size analyses demonstrate that duloxetine 40 mg has minimum efficacy, and that duloxetine 60–120 mg/day is effective in the treatment of patients with MDD. An initial dose less than 60 mg/day might provide better tolerability for some patients diagnosed with MDD.

The Importance of Effect Sizes in the Interpretation of Research

The Importance of Effect Sizes in the Interpretation of Research

Bibliometric Reflections on the First Decade of Human Dimensions of Wildlife

Volume 1, number 1 of Human Dimensions of Wildlife (HDW) was published in 1996. This article reviews the first 10 years of HDW using bibliometric analysis. Bibliometrics sheds light on the process of written communication by counting and analyzing publications within a discipline. The findings highlighted some encouraging trends. First, the number of international contributions nearly doubled from 1996–2000 to 2001–2005. Second, a wide range of wildlife-related issues (e.g., consumptive activities, nonconsumptive activities, preservation efforts) were examined. Third, most major social science disciplines were represented in all 10 volumes of HDW. Fourth, the broad range of concepts examined in HDW articles is consistent with the founding objective of promoting the application of social science theories to fisheries and wildlife management. Finally, the use of effect sizes in addition to null hypothesis significance tests increased from 1996–2000 to 2001–2005, indicating that HDW authors are practicing sound statistical practices.

Practical Significance:The Use of Effect Sizes in School Counseling Research

In response to the increasing call by professional organizations, journal editors, and statisticians to include not only derived significance levels from quantitative statistical procedures but also measures of effect sizes (ES), this article first provides a rationale for school counseling-related researchers to include these key indices in their studies. We contend that with this information, readers are better able to contextualize and properly interpret research findings and conclusions. Second, the three major ES families and the indices associated with each one are reviewed. Next, the influence of various research designs and differing sample sizes on the size of these effects is discussed, followed by an introduction to ES computation, reporting, and interpretation, particularly as these issues relate to school counseling settings. Practical examples, ES summary tables, and supplementary resources are overviewed as well.

A Closer Look at Effect Sizes and Their Relevance to Health Education

A number of academic disciplines are engaged in scholarly discussions regarding statistical practice reform, particularly the use of effect sizes. Health education must stimulate a similar conversation by adopting strategies for generating, reporting, and interpreting effect size estimates for various statistical analyses within journal articles. The purpose of this article is to demonstrate the practical applications of effect size reporting and interpretation in health education research. Congruent with previous recommendations in the American Journal of Health Education, this article will provide examples and techniques used for effect size reporting that educate the researcher and practitioner, thus improving the scholarship of health education publications. Effect size reporting should become the rule for health education and concerted efforts should be made to equip researchers and practitioners with the proficiency to perform this task effectively. Such skill building will increase the scholarship and readability of health education research.

Practical Significance: The Use of Effect Sizes in School Counseling Research

In response to the increasing call by professional organizations, journal editors, and statisticians to include not only derived significance levels from quantitative statistical procedures but also measures of effect sizes (ES), this article first provides a rationale for school counseling-related researchers to include these key indices in their studies. We contend that with this information, readers are better able to contextualize and properly interpret research findings and conclusions. Second, the three major ES families and the indices associated with each one are reviewed. Next, the influence of various research designs and differing sample sizes on the size of these effects is discussed, followed by an introduction to ES computation, reporting, and interpretation, particularly as these issues relate to school counseling settings. Practical examples, ES summary tables, and supplementary resources are overviewed as well.

Comparing SF-36 scores across three groups of women with different health profiles

The widespread use of the Medical Outcomes Study (MOS) 36-item Short-Form Health Survey (SF-36) facilitates the comparison of health-related quality of life (HRQL) across independent studies. To compare the scores of eight scales and two summary scales of the SF-36 across participants in the Women's Healthy Eating and Living (WHEL) trial, the Women's Health Initiative-Dietary Modification trial (WHI-DM), and the MOS, and to illustrate the use of effect sizes for interpreting the importance of group differences. WHEL and WHI-DM are both multi-center dietary interventions; only data from the UC Davis sites were used in our study. WHEL participants had a recent history of breast cancer, WHI-DM participants were healthy, postmenopausal women, and women in the MOS had a history of hypertension, diabetes, heart disease, or depression. General linear models were used to identify statistically significant differences in scale scores. Meaningful differences were determined by effect sizes computed using a common within-group standard deviation (SD) and SDs from normative data. After adjusting for age and marital status, SF-36 scores for the WHI-DM and WHEL samples were similar and both had statistically significantly higher scores than the MOS sample. Relative to the WHEL or WHI-DM studies, MOS scores for scales related to the physical domain were clearly meaningfully lower whereas scale scores related to the mental health domain were potentially meaningfully lower. The HRQL of breast cancer survivors is comparable to that of healthy women and better than that of women with chronic health conditions, particularly with respect to physical health. This study illustrated the use of ranges of effects sizes for aiding the interpretation of SF-36 scores differences across independent studies.

Validation des échelles d'évaluation en médecine physique et de réadaptation : comment apprécier correctement leurs qualités psychométriques

The psychometric properties of a rating scale are, unfortunately, badly assessed in practice. Each property is thoroughly studied. Validity has different facets: content, criterion, construct, face validity. Each facet is accurately defined and the experimental and statistical methods used are explained in detail. Reliability is assessed with use of the intraclass correlation coefficient and the Bland and Altman method. Guidelines are given for choosing, in a specific experimental situation, the appropriate coefficient. Responsiveness is evaluated with use of effect size and standardized response mean. Finally practical recommendations are given.

The Use of Effect Sizes in Music Therapy Research

ABSTRACT: Music therapy outcome research frequently involves comparisons between groups on continuous scales of psychological constructs. Describing such differences only through statistical tests reduces the existing complexity and creates an artificial and potentially misleading dichotomy. The interpretation of the magnitude of a difference found on a psychological scale is often not straightforward, but can be greatly facilitated and improved by using effect sizes. Cohen's (1988) benchmarks allow for intuitive judgements and enable comparisons between different scales. An example of a study of the effects of music therapy on self-esteem shows how the use of effect sizes can change the interpretation of a research result. Effect sizes have important applications in many fields of music therapy research, including primary studies as well as meta-analyses, and planning of research as well as reporting of research results. Because of their intuitiveness, they may help to bridge the gap between research and clinical practice. There are many different types of research designs and strategies which are applied in music therapy research, including experimental to descriptive forms of research. Experimental and quasi-experimental research on music therapy outcomes frequently involves a comparison between two groups, such as an experimental group and a control group. Different characteristics can be chosen to represent the outcomes of the groups, including, for example, frequency counts of behaviors and psychological constructs. The latter type of outcome is most commonly measured on a continuous scale. Examples of such scales include severity of psychiatric symptoms, degree of social functioning, and level of self-esteem, among others. The purpose of outcome research that involves comparisons between groups is to draw conclusions about the clinical effects of a treatment procedure (e.g., a music therapy program), when compared with a different procedure (e.g., no treatment, standard care, verbal therapy, or another music therapy program). Whenever the means of two groups are compared, there are various ways of describing the difference. The first question that can be addressed is the size of the difference between these two groups. This question is related to the clinical relevance of the difference (or the effect of treatment) and can be answered using an effect size (ES) calculation. If the two groups represent random samples drawn from larger populations, the second question that can be asked is whether one can be sure that there is a difference between the populations from which our sample was drawn. This question refers to inferences that can be made of the generalizability of the results from a representative sample to a larger population, and can only be answered indirectly with a test of statistical significance, which tells how likely-or unlikely-the sample would be drawn in the absence of a difference between the populations. Researchers in the social sciences have often addressed differences between groups only in terms of inferential statistics, without undertaking a descriptive analysis of the differences in their sample, possibly because they were not aware of the potential for calculating the size of an effect (Cohen, 1988). Problems with Statistical Tests There are several problems with the exclusive use of inferential statistics. First, since tests of significance are designed to accept or reject a null hypothesis, they lead to a dichotomous decision, such as between yes and no, or between and white, suggesting that there is either an effect or no effect. This is inappropriate for research questions where it is relevant to what degree a null hypothesis may be wrong. To stay with this scenario, deciding between black and white ignores the many shades of gray that may exist in between. second, the statistical indices that a test of significance provides are, by nature, not useful (and not intended) as descriptive statistics because they depend on effect size and sample size. …

Comparing and combining studies of bronchial responsiveness

Background: There is no standardised protocol for the measurement of bronchial responsiveness. Results from different studies are difficult to compare and combine. Methods: Analyses are divided between those of a...

Psychopathology through a life span-genetic prism.

The variation observed in individual differences for normal and psychopathological behaviors has genetic factors as a major contributor at the most distal end of a complex gene-to-behavior pathway. Research into the etiologies of such major mental diseases as schizophrenia is facilitated by adopting the approach used for complex adaptive systems as pursued by those who study coronary artery disease and diabetes. Putative risk factors for developing the liabilities to the major disorders can be inferred from population genetic strategies using twins, families, and adoptees. Weights to indicate the relative importance of such risk factors require a perspective from the use of effect sizes and odds ratios so as to make the most efficient use of scarce resources. The challenge to the field of psychology is to join in with cross-disciplinary ventures and to adapt to rapid changes with innovations in research strategies.

Meta-analyses of prospective studies on coronary heart disease, type A personality, and hostility

Background: Several large-scale prospective studies have failed to find an association between Type A personality (TAP) or hostility and coronary heart disease (CHD). The existing meta-analyses on this topic have several shortcomings, such as a selective search of the literature, the failed use of effect sizes, and methodological flaws. Methods: The present meta-analyses cover all prospective studies until the end of 1998. The correlation coefficient r is used as effect size to yield information on the population effect size R and variance. Several analyses have been carried out to stratify for disease endpoints, samples (healthy population, CHD patients), and methods used to determine TAP or hostility. Results: The population effect size (weighted average of all correlation coefficients) for TAP and CHD is R=0.003 ( n=74,326, P=0.213), and for hostility and CHD R=0.022 ( n=15,038, P=0.003). Conclusions: The population effect size for TAP and CHD is not significant. Hostility yields a significant association with CHD; however, the effect size is so low that it has as yet no practical meaning for prediction and prevention.

Research design and statistics in biomechanics and motor control

Biomechanics and motor control researchers measure how the body moves and interacts with its environment. The aim of this review paper is to consider some key issues in research methods in biomechanics and motor control. The review is organized into four sections: proposing, conducting, analysing and reporting research. In the first of these, we emphasize the importance of defining a worthy research question and of planning the study before its implementation to prevent later difficulties in the analysis and interpretation of data. In the second section, we cover selection of trial sizes and suggest that using three trials or more may be beneficial to provide more ‘representative’ and valid data. The third section on analysis of data concentrates on effect size statistics, qualitative and numerical trend analysis and cross-correlations. As sample sizes are often small, the use of effect size is recommended to support the results of statistical significance testing. In using cross-correlations, we recommend that scatterplots of one variable against the other, with the identified time lag included, be inspected to confirm that the linear relationship assumption underpinning this statistic is met and, if appropriate, that a linearity transformation be applied. Finally, we consider important information related to the issues above that should be included when reporting research. We recommend reporting checks or corrections for violations of underpinning assumptions, and the effect of these checks or corrections, to assist in advancing knowledge in biomechanics and motor control.

Factors That Influence Effect Size in15O PET Studies: A Meta-analytic Review

The PET literature is growing exponentially, creating a need and an opportunity to perform a meta-analytic review consolidating the published information. This study describes the use of effect size as an index in PET studies and discusses how this measure can be used for comparing findings across studies, laboratories, and paradigms. In comparing studies across laboratories it is essential to know how the methods employed affect the results and conclusions drawn. This study also compared effect size for two different methods of tracer delivery in15O PET studies ([15O]H2O bolus injection versus inhalation of [15O]CO2), whether averaged versus single-scan conditions were used, and the data analytic strategy employed. The effect sizes observed across studies were consistently large with a median effect size of 8.55, indicating that the phenomena investigated in15O PET studies are strong. The largest peak activation reported in a study was found to be affected by variability in sample size, data analytic strategy, and repeat versus single-scan conditions. However, the impact of these factors was not examined on smaller or less intense peaks. Minimal standards for reporting statistical results are discussed.

Dichotic listening, handedness, and reading ability: A meta‐analysis

A meta‐analysis was used to aggregate data regarding the influence of selective attention, handedness, and reading ability on the right‐ear advantage in dichotic listening. Data from 15 independent dichotic studies that employed consonant‐vowel stimuli, free‐recall and directed‐attention conditions, and left‐and right‐handed children (age range 6 to 12 years, IQ > 85) classified as good and poor readers determined the sample. The statistical aggregation of research results based on the use of effect sizes and one‐sample tests indicated that younger (6 years, 0 months to 8 years, 11 months) and older (9 years, 0 months to 12 years, 11 months) good readers and older poor readers shifted attention and thus overcame the right ear advantage bias, whereas younger poor readers shift attention only in the directed‐right condition. Effect size analyses (with individuals combined for age) also indicated that left‐ and right‐handed good readers do not shift attention across directed conditions, but attentional shifting was found for left‐ and right‐handed poor readers. The meta‐analysis clearly demonstrates an interaction among age, reading ability, and handedness on dichotic listening outcome. Differences in patterns of ear report among these groups of children are suggestive of a combination of varied functional lateralization and attentional strategies employed during dichotic consonant‐vowel tasks.

Determining the Efficacy of Intervention: The Use of Effect Sizes for Data Analysis in Single-Subject Research

Abstract Use of the effect size as a descriptive statistic for single-subject research is presented. A brief review of visual and statistical analysis techniques commonly used in single-subject methods is provided, and the limitations of each are noted. Effect sizes are presented as statistics that can augment the interpretation of results as well as provide additional information about the effectiveness of interventions. Four types of treatment effects are presented, with corresponding case studies illustrating the computation and interpretation of the effect size for each. An appendix includes the case study data and a sample computer program for computing the effect sizes described.

Improving the Understanding of Statistical Significance

The use of effect size as an approach to improve the communication of the clinical or policy significance of a statistically significnat t-test was investigated with three groups of statistics students Presenting only the values of t, df, and probability resulted in an overestimation of the difference between the means of a control group and the treatment group Including a statement about effect size and the percentile rank of the mean of the treatment group with respect to the control group, improved the students' accuracy (p < .05). Providing effect size information enabled undergraduates to nearly match the accuracy of graduate students who were not given effect size information.