Effect Size Statistics Research Articles

Student and faculty gender in ratings of university teaching quality

The aim of this study is to test the assertion that student and faculty gender, together with their possible interaction, have little or almost no real effect on student ratings of university teaching quality, taking into account the diversity of academic disciplines as a modulating variable. A sample of 1,304 student ratings from a university in Madrid, Spain, was selected out of a total of 236,862. The Complutense University Teachers Evaluation Questionnaire was administered to students at the Complutense University of Madrid for assessing teaching quality. This instrument operationalizes two main dimensions in evaluation of teaching in higher education: Teaching Competence and Motivational and Interactive Skills. When the results were analyzed, taking into consideration both possible statistical significance and effect size, they supported the proposed assertion. These results may have implications on certain areas of higher education.

Debunking the myth of the “highly significant” result: Effect sizes in gifted education research

The use of statistical effect sizes in educational research is frequently recommended, especially with the recent emphasis on meta‐analysis of research results. The purpose of this article is to describe the utility of effect size reporting and interpretation and to investigate the use of statistical effect sizes in research on giftedness. Research articles published in a five‐year span in Journal for the Education of the Gifted, Roeper Review, Gifted Child Quarterly, and a collection of journals not directly associated with gifted education were content analyzed with respect to the reporting of effect sizes. Results suggest that effect sizes generally are not included in research articles, with results consistent across journals.

Outcome, attrition, and family-couples treatment for drug abuse: a meta-analysis and review of the controlled, comparative studies.

This review synthesizes drug abuse outcome studies that included a family-couples therapy treatment condition. The meta-analytic evidence, across 1,571 cases involving an estimated 3,500 patients and family members, favors family therapy over (a) individual counseling or therapy, (b) peer group therapy, and (c) family psychoeducation. Family therapy is as effective for adults as for adolescents and appears to be a cost-effective adjunct to methadone maintenance. Because family therapy frequently had higher treatment retention rates than did nonfamily therapy modalities, it was modestly penalized in studies that excluded treatment dropouts from their analyses, as family therapy apparently had retained a higher proportion of poorer prognosis cases. Re-analysis, with dropouts regarded as failures, generally offset this artifact. Two statistical effect size measures to contend with attrition (dropout d and total attrition d) are offered for future researchers and policy makers.

Use of goal attainment scaling in measuring clinically important change in cognitive rehabilitation patients

Measuring the effectiveness of cognitive rehabilitation programs poses both conceptual and practical challenges. We compared several standardized outcome measures with goal attainment scaling (GAS) to assess their sensitivity to changes in health status in patients undergoing cognitive rehabilitation. GAS is a measurement approach that accommodates multiple individual patient goals, and has a scoring system which allows for comparisons between patients. Forty-four patients were evaluated. GAS yielded a mean 4.4 goals per patient. The mean gain in the GAS score was compared with the change in the Rappaport Disability Rating Scale, the Kohlman Evaluation of Daily Living Skills, the Milwaukee Evaluation of Daily Living, the Klein-Bell elimination scale and mobility scale, the Instrumental Activities of Daily Living Scale, and the Spitzer Quality of Life Index. Using a relative efficiency statistic, GAS proved more responsive than any other measure. The effect size statistic also demonstrated greater responsiveness to change with GAS compared with standard measures. GAS shows promise as a responsive measure in cognitive rehabilitation. This study replicates a similar study of GAS in frail elderly patients, suggesting that individualized measures may have broad merit in evaluating rehabilitation programs.

How interchangeable are different estimators of effect size?

The computation of effect sizes is a key feature of meta-analysis. In treatment outcome meta-analyses, the standardized mean difference statistic on posttest scores (d) is usually the effect size statistic used. However, when primary studies do not report the statistics needed to compute d, many methods for estimating d from other data have been developed. Little is known about the accuracy of these estimates, yet meta-analysts frequently use them on the assumption that they are estimating the same population parameter as d. This study investigates that assumption empirically. On a sample of 140 psychosocial treatment or prevention studies from a variety of areas, the present study shows that these estimates yield results that are often not equivalent to d in either mean or variance. The frequent mixing of d and other estimates of d in past meta-analyses, therefore, may have led to biased effect size estimates and inaccurate significance tests.

Responsiveness of functional status in low back pain: a comparison of different instruments

This study compares the responsiveness of three instruments of functional status: two disease-specific questionnaires (Oswestry and Roland Disability Questionnaires), and a patient-specific method (severity of the main complaint). We compared changes over time of functional status instruments with pain rated on a visual analog scale. Two strategies for evaluating the responsiveness in terms of sensitivity to change and specificity to change were used: effect size statistics and receiver-operating characteristic method. We chose global perceived effect as external criterion. A cohort of 81 patients with non-specific low back pain for at least 6 weeks assessed these measures before and after 5 weeks of treatment. According to the external criterion 38 patients improved. The results of both strategies were the same. All instruments were able to discriminate between improvement and non-improvement. The effect size statistics of the instruments were higher in the improved group than in the non-improved group. For each instrument the receiver-operating characteristic curves showed some discriminative ability. The curves for the Roland Questionnaire and pain were closer to the upper left than the curves for the other instruments. The sensitivity to change of the rating of Oswestry Questionnaire was lower than that of the other instruments. The main complaint was not very specific to change. The two strategies for evaluating the responsiveness were very useful and appeared to complement each other.

Effects of physical exercise on cognitive functioning in the elderly

In this paper, we summarize the effects of aging on physiological and cognitive processes in elderly individuals, discuss the ameliorative effects of physical exercising on these processes, and present a review of previous reports of the effects of physical exercise on cognitive functioning in the elderly. We also underscore issues concerning research design and methodology. Thirteen of 18 studies provided sufficient information for calculation of statistical power and effect sizes. Although these studies included a total of 87 statistical tests, the power exceeded. 4 in only 10 instances. Despite inadequate power, effects of exercise were strong enough to yield significant results (<.05) in 23 of these tests. Of the 87 effect sizes determined by using pre- to postassessment data for the exercise group, 56 fell within the very small to medium range (.04 to .49), and 15 fell within the medium to large range (.05 to 1.59). We conclude that exercise promotes modest changes in cognitive functioning in the aged. Given demonstrated improvements in both physical and psychological health, a sound empirical basis exists for prescription of exercise for older adults.

The effect of sugar on behavior or cognition in children. A meta-analysis.

To examine the effects of sugar on the behavior or cognition of children by using meta-analytic techniques on reported studies. Studies were identified through a literature search of the MEDLINE and PsychINFO databases and the authors' files using sugar, sucrose, and attention deficit disorder as the search terms. Studies were required to (1) intervene by having the subjects consume a known quantity of sugar, (2) use a placebo (artificial sweetener) condition (3) blind the subjects, parents, and research staff to the conditions; and (4) report statistics that could be used to compute the dependent measures effect sizes. Variables included publication year, study setting, subject type and number, gender, age, sugar and placebo type and dose, prior dietary condition, measurement construct, means and SDs for the sugar and placebo conditions, and direction of effect. Sixteen reports met the inclusion criteria for a total of 23 within-subject design studies. The weighted mean effect size and related statistics for each of the 14 measurement constructs revealed that although the range for these means was from -0.14 for direct observations and up to +0.30 for academic tests, the 95% confidence interval for all 14 mean effect sizes included 0. The meta-analytic synthesis of the studies to date found that sugar does not affect the behavior or cognitive performance of children. The strong belief of parents may be due to expectancy and common association. However, a small effect of sugar or effects on subsets of children cannot be ruled out.

Statistical power in operations management research

AbstractThis paper discusses the need and importance of statistical power analysis in field‐based empirical research in Production and Operations Management (POM) and related disciplines. The concept of statistical power analysis is explained in detail and its relevance in designing and conducting empirical experiments is discussed. Statistical power reflects the degree to which differences in sample data in a statistical test can be detected. A high power is required to reduce the probability of failing to detect an effect when it is present. This paper also examines the relationship between statistical power, significance level, sample size and effect size. A probability tree analysis further explains the importance of statistical power by showing the relationship between Type II errors and the probability of making wrong decisions in statistical analysis. A power analysis of 28 articles (524 statistical tests) in the Journal of Operations Management and in Decision Sciences shows that 60% of empirical studies do not have high power levels. This means that several of these tests will have a low degree of repeatability. This and other similar issues involving statistical power will become increasingly important as empirical studies in POM study relatively smaller effects.

Use of a common language effect size statistic

Use of a common language effect size statistic

The halstead‐reitan neuropsychological test battery for older children: The effects of age versus clinical status on test performance

A meta‐analysis of reference group data for the Halstead‐Reitan Neuropsychological Test Battery for Older Children (HRB‐OC), 9 to 14 years of age, is presented, using the effect size statistic described by Glass (1978). The distance from a pooled normal mean is calculated for published reference data from two groups of normal 9‐year‐old children and six groups of children with compromised neuropsychological status. Results indicate that using age‐referenced normative data may significantly enhance interpretive accuracy with the HRB‐OC, especially for children at the ends of the HRB‐OC age spectrum. The ability of the HRB‐OC tests to separate children by age and by clinical status varies across tests and across samples.

Use of goal attainment scaling in measuring clinically important change in the frail elderly

The selection of appropriate outcome measures is important in evaluating specialized geriatric programs, but how the various measures compare, and which are most appropriate, are matters still largely unexplored.We compared several outcome measures, including goal attainment scaling, to assess their sensitivity to changes in the health status of frail elderly patients admitted to two geriatric medicine wards. GAS is a measurement approach which accommodates multiple individual patient goals, and has a scoring system which allows for comparisons between patients.Forty-five patients (mean age 81 years, 30 females) received comprehensive assessments. GAS yielded a mean 5 goals per patient. The mean gain in the GAS score was 22 points (SD = 7) which was compared with the change in the Barthel Index (r = 0.59), the Functional Independence Measure (r = 0.45), the Physical Self-Maintenance Scale (r = 0.54), the Katz Activities of Daily Living Index (r = 0.49) and the Spitzer Quality of Life Index (r = 0.38). The inter-rater reliability of scoring the GAS follow-up guides was 0.91. Using a relative efficiency statistic, GAS proved more efficient than any other measure. The effect size statistic also demonstrated an increased responsiveness to change of GAS compared with standard measures. GAS is an individualized measurement approach which shows promise as a responsive measure in frail elderly patients.

A common language effect size statistic.

Some of the shortcomings in interpretability and generalizability of the effect size statistics currently available to researchers can be overcome by a statistic that expresses how often a score sampled from one distribution will be greater than a score sampled from another distribution. The statistic, the common language effect size indicator, is easily calculated from sample means and variances (or from proportions in the case of nominal-level data). It can be used for expressing the effect observed in both independent and related sample designs and in both 2-group and n-group designs

A common language effect size statistic.

A common language effect size statistic.

Reproducibility and responsiveness of health status measures statistics and strategies for evaluation

Before being introduced to wide use, health status instruments should be evaluated for reliability and validity. Increasingly, they are also tested for responsiveness to important clinical changes. Although standards exist for assessing these properties, confusion and inconsistency arise because multiple statistics are used for the same property; controversy exists over how to measure responsiveness; many statistics are unavailable on common software programs; strategies for measuring these properties vary; and it is often unclear how to define a clinically important change in patient status. Using data from a clinical trial of therapy for back pain, we demonstrate the calculation of several statistics for measuring reproducibility and responsiveness, and demonstrate relationships among them. Simple computational guides for several statistics are provided. We conclude that reproducibility should generally be quantified with the intraclass correlation coefficient rather than the more common Pearson r. Assessing reproducibility by retest at one-to-two week intervals (rather than a shorter interval) may result in more realistic estimates of the variability to be observed among control subjects in a longitudinal study. Instrument responsiveness should be quantified using indicators of effect size, a modified effect size statistic proposed by Guyatt, or the use of receiver operating characteristic (ROC) curves to describe how well various score changes can distinguish improved from unimproved patients.

Union is Strength: A Consumer's View of Meta-Analysis

Meta-analyses are a textbook author's delight. They can alert authors to the existence of a literature, quantitatively synthesize results, and mitigate overemphasis on vivid but misleading studies. For these benefits text authors pay a price: The need to simplify impedes scrutiny of individual studies and draws attention away from moderating variables and toward the statistical bottom line-the mean effect size.

Computational approaches to analytical reasoning: a comparative analysis

The water maze is one of the most widely employed spatial learning paradigms in the cognitive profiling of genetically modified mice. Oftentimes, tests of reference memory (RM) and working memory (WM) in the water maze are sequentially evaluated in the same animals. However, critical difference in the rules governing efficient escape from the water between WM and RM tests is expected to promote the adoption of incompatible mnemonic or navigational strategies. Hence, performance in a given test is likely poorer if it follows the other test instead of being conducted first. Yet, the presence of such negative transfer effects (or proactive interference) between WM and RM training in the water maze is often overlooked in the literature. To gauge whether this constitutes a serious concern, the present study determined empirically the magnitude, persistence, and directionality of the transfer effect in wild-type C57BL/6 mice. We contrasted the order of tests between two cohorts of mice. Performance between the two cohorts in the WM and RM tests were then separately compared. We showed that prior training of either test significantly reduced performance in the subsequent one. The statistical effect sizes in both directions were moderate to large. Although extended training could overcome the deficit, it could re-emerge later albeit in a more transient fashion. Whenever RM and WM water maze tests are conducted sequentially in the same animals – regardless of the test order, extra caution is necessary when interpreting the outcomes in the second test. Counterbalancing test orders between animals is recommended.

Patient suitability and outcome in short-term individual psychotherapy.

The effects of short-term (time-limited), dynamically oriented individual therapy and the patient characteristic quality of object relations (QOR) were studied in a controlled clinical trial involving 8 experienced therapists and 144 psychiatric outpatients. A comprehensive set of outcome scores was monitored at 5-month intervals including a follow-up assessment. A strong treatment effect that was maintained at follow-up was found. The effect was evident in terms of statistical significance, effect size, and clinical significance. Evidence for an additive effect of the combination of treatment and QOR was also found. The best results were attained by high QOR therapy patients, and the worst results were attained by low QOR control patients. The results were suggestive of an optimal match between type of patient and type of therapy.

Designing Field Experiments for Management Research Outside North America

Research on foreign and international management has progressed substantially [1] since Roberts' s 1970 review [2], which outlined many of the weaknesses inherent in cross-national or cross-cultural studies. Although Adler [3] has demonstrated that Corporate activity has internationalized faster than the publishing of articles in American management journals, one can see the progress in the research, from largely anecdotal reports in the 1950s to studies using fairly sophisticated methodologies in the 1980s [4]. Researchers are attempting to compensate, through increased rigor, for many weaknesses previously identified [5-9]. This increased sophistication of foreign and international management research is welcome; but, as in all research, the investigator choosing from a variety of research designs must consider where and when each is appropriate. My purpose here is to examine the potential of experimental field studies in this context. I shall first describe experimental research designs in general and some advantages and disadvantages associated with those designs, then consider some potential problems in foreign experimental field studies and the issue of statistical controlsa levels, power, and effect size. An ongoing research project in the Caribbean will be used to illustrate points throughout.

Statistical significance, power, and effect size: A response to the reexamination of reviewer bias.

Statistical significance, power, and effect size: A response to the reexamination of reviewer bias.