EFFECT SIZE UNDERESTIMATES THE EFFECTS OF INTERVENTIONS AMONG OLDER PEOPLE WITH SEVERE PHYSICAL OR COGNITIVE IMPAIRMENTS?

Abstract

To the Editor: The use of effect sizes for comparison of the effects of various studies could potentially underestimate the effect among older people with severe physical or cognitive impairment. In a Cochrane Review about strength training in older adults, Standardized Mean Difference (SMD) was used to estimate the effect on lower-limb strength, calculated as the difference in means (between the intervention and the control group) of the outcome divided by the standard deviation of the outcome.1 Using a calculation where the effect is divided by the variation in ability between participants may underestimate the effect in studies which have broad inclusion criteria (often the case in studies on older people with severe impairments) since the same change over time in the absolute value of an outcome will produce a higher effect size in a more homogeneous group, as is to be expected if the participants are more healthy. This may, in the worst case, lead to incorrect treatment recommendations for older people with physical and cognitive impairments. In addition, differences in baseline values are likely to be seen in intervention studies among older people, despite their being randomized, since the studies are often relatively small. However, this seems not to have been considered in the Cochrane Review for all included studies. For example in a study by Skelton et al., the effect regarding knee extensor strength, as presented in the review, was negative although the improvement from baseline to follow-up was considerably greater in the intervention group than in the control group (+27% vs +3%).2 The calculations of SMD for this study seem to be based in the review on the unadjusted outcome values at the follow-up and do not take into consideration the fact that baseline values were lower in the intervention group than in the control group. An alternative way of calculating the effect size is to use the change in the outcome from baseline to follow up (both the difference between groups and the standard deviation), based on estimated marginal means for the outcome at the follow-up adjusted for baseline values.3 This approach takes into account differences between groups at baseline and variations in ability between participants. Hopefully, in future, a consensus will be reached concerning how to calculate effect sizes in randomized trials. For the present, it is recommended that effect sizes are presented but the definitions in the CONSORT statement,4 e.g., for continuous outcome “difference in means,” need to be clarified to ensure that the calculation is made in the same way in all studies. Financial Disclosures: None. Author Contributions: I am the sole author of this letter. Sponsors' Role: None.