In a comment directed at the fallacies of meta-analyses, Kazrin,Durac, and Agteros (1979) imagined a process which they termed ameta-meta-analysis. Stating that “the strength of the meta-meta-analysis, like the unfortunate person stranded on a desert island,is that it will look at anything” (p. 397), they whimsically arguedhow this process, directed at psychotherapy effectiveness, utilised850,000 therapists and 2,000,000 clients. Such tongue in cheekwarnings regarding the misuse of meta-analyses while evaluatingpsychotherapy effectiveness should be taken seriously. Indeed,looking at the inclusion and exclusion criteria within meta-analyses, Barlow (2010) warns us of cherry-picking studies statingthat “it is easy tosit backand ‘pick-off’ any newstudyand concludethat it’snot‘perfect’” (p. 16). For this reason, Barlow argues, weneed to be very cautious when similar meta-analyses includedifferent studies and come to different conclusions.Lee and Cuijpers (2013), in their meta-analysis of ‘trials’ lookingpurely at the utility of eye movements in EMDR, concatenate eyemovements, negative imagery, autobiographical memories con-taining distress, and randomised experimental study or treatmentwith patients or students to place their inclusion criteria. Havingset-up the meta-analysis exclusion criteria they accordinglyexclude large numbers of studies which tested this exact phe-nomenon for various reasons of technicality or preference (e.g.,Devilly & Brown, 2011; Devilly & Spence, 1999; Jensen, 1994;Merckelbach, Hogervorst, Kampman, & de Jongh, 1994; Pitmanet al., 1996; Rothbaum, Astin, & Marsteller, 2005; Taylor et al.,2003; to name just a few) or cherry-picking (or combining)incompatible outcome measures when the studies have beenincluded (e.g., Dunn, Schwartz, Hatfield & Weigele, 1996 present awealth of physiological data rather unsympathetic to EMDR, whichwas not factored into the analysis). The authors then wish toinclude two of the studies twice each in the same meta-analysis(incidentally one of these twice counted studies was conductedby the first author of the meta-analysis). One of the main reasonsforameta-analysisistoreducecommonmethodvariancebiasfromindividual research groups. Whether this actually increases or de-creases the derived effect size is irrelevant in comparison to theprinciple of introducing error from the same study twice. Besidesnegating this benefit, the current approach also requires moreclarificationonhowonesetofconfidenceintervalsaroundaneffectsize mean was computed for each individual study from multipleoutcome measures (their Figure 2 and 3). We assume that the in-terval around this average was computed based upon total samplesize rather than the number of measures combined or a multipli-cation of number of measures by sample size or some other esti-mate of error.Lee’s twice counted study utilised students, as opposed to pa-tients, as did some other studies included in the meta-analysis andalso, by their own admission, deviated extensively from the EMDRprotocol. We are unclear why one would accept this study andsimilar studies, yet not include large trials which compared expo-sure with and without eye movements (i.e., PE vs EMDR). If indeedthe argument is that it is the eye movements which make thedifference, then excluding them appears quite inconsistent.Lee & Cuijpers obtain an effect size which they claim is signifi-cant. The fact that an effect size of 0.27 is significant in this meta-analysis (when controlling for two extreme outliers) from 12studies(or13ifweincludeLee’sstudytwice)wouldsuggestthatwehave complete agreement amongst these studies. However, with 5out of the 12 studies finding effect sizes in exactly the opposite di-rection or finding a zero sum result, and with us being aware ofexcluded studies also finding opposite results, we remain uncon-vincedbythismeta-analysis.Excludingstudieswhichcontrolledforplacebo effects (such as tapping and clicking) and not includingstudies unless it focussed on a negative autobiographical memory(i.e., excluding studies which concentrated on trauma words, anxi-ety, etc), and at the same time excluding any comparison groups
Read full abstract