Critical Appraisal of Individual Trials?

Abstract

TO THE EDITOR: Given all that can go wrong even in an ostensibly good clinical trial (1,2), Gluud et al. (3) are quite correct that appraisal of individual trials is always necessary, irrespective of the place of publication. And yet they provide no indication as to why they did not, in fact, critically appraise the trials they claimed to, at least with respect to allocation concealment. Accepting that central independent unit, identical containers, sealed envelopes, or similar can acceptably conceal the future allocations is rather uncritical. The steps listed all attempt to eliminate the direct observation of the identity of upcoming treatment allocations, although in practice, they meet with varying success. While the success of these steps is far from guaranteed, in fact, there is a far more sinister reason for skepticism. Even if these steps were to ensure that there could be no direct observation of the identity of upcoming treatment allocations, this would still be insufficient to ensure allocation concealment. This is because a researcher can predict the upcoming allocations based on the knowledge of both the prior allocations and patterns in the allocation sequence due to restrictions on the randomization, such as permuted blocks (4,5). This is true especially in unmasked trials, but even in ostensibly masked trials the possibility exists that some prior allocations will be unmasked. This means that any critical appraisal of allocation concealment needs to consider the extent of unmasking and the specific restrictions used for the randomization, and none of the steps mentioned by Gluud et al. (3) address these important considerations at all. The result is the unfortunate state of affairs in which the reader has no idea how many of the trials considered actually had successful allocation concealment. This has to throw into question the validity of the findings, not only of Gluud et al. (3), but also of the trials they considered, because without allocation concealment, selection bias may create the illusion of a treatment effect when in fact one does not exist (5).