Using Standard Studies to Interpret the Scores from Scales for Assessing the Internal Validity of Randomized Controlled Trials

Abstract

Listen

Translate article

Objective A large number of tools for assessing the quality of randomized controlled trials are available; however, users have little guidance as to whether a given score represents high or low validity. The purpose of this study is to explore the use of studies identified as having high-internal validity, referred to as the standard studies, to interpret internal validity scores from studies with unknown internal validity. Methods The standard studies were identified by locating 6 candidate studies reporting the findings of randomized controlled trials from the Journal of American Medicine Association or the New England Journal of Medicine and scoring the studies using 2 scales, the Jadad scale (high score = 5; low = 0) and an internal validity information scale (IVI; high score = 70; low = 0). The 2 studies with the highest average rank were chosen as the standard studies. To determine if the standard studies facilitate interpretation of internal validity scores, 11 randomized controlled trials of garlic for the treatment of hypercholesterolemia were scored using both the Jadad and IVI scales and differences in scores resolved by consensus. The total mean scores were compared with the standard-study mean using a 1-sample t test; the interpretation was based on the comparison. Results The mean-internal validity score for the 2 standard studies was 4.5(90%) for the Jadad scale and 64 (91%) for the IVI scale. For the garlic studies, the Jadad scores ranged from 3 to 5 (60% to 80%), the IVI scores from 15 to 59 (21% to 84%); the mean scores were 4 (standard deviation = 0.9; 80%) and 37(53%), respectively. On the IVI scale, the garlic studies had lower internal validity (P < 0.001), a finding substantiated by the low scores for addressing withdrawals, blinding, and adherence. On the Jadad scale, the scores were comparable with the score for the standard studies (P = 0.093). Conclusions The strategy of using the standard studies to interpret the scores for the IVI scale worked well; the comparison indicated that, overall, the garlic studies were of low-internal validity. With the Jadad scale, the strategy resulted in erroneous conclusions that the garlic studies had relatively high-internal validity; however, it did indicate that the Jadad scale is limited and may not identify studies with very low-internal validity. The results of this study suggest that scales used to assess internal validity should be tested on studies with a range of validity, so the functional characteristics of the scale can be identified.