Systematic mixed-methods reviews are not ready to be assessed with the available tools

Abstract

Systematic reviews are central to knowledge translation [1Popay J. Roberts H. Methodological issues in the synthesis of diverse sources of evidence.in: Popay J. Moving beyond effectiveness in evidence synthesis: methodological issues in the synthesis of diverse sources of evidence. National Institute for Health and Clinical Excellence, London, UK2006: 1-10Google Scholar, 2Pluye P. Gagnon M.-P. Griffiths F. Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in Mixed Studies Reviews.Int J Nurs Stud. 2009; 46: 529-546Abstract Full Text Full Text PDF PubMed Scopus (610) Google Scholar, 3Dixon-Woods M. Bonas S. Booth A. Jones D.R. Miller T. Sutton A.J. et al.How can systematic reviews incorporate qualitative research? A critical perspective.Qual Res. 2006; 6: 27-44Crossref Scopus (476) Google Scholar] and have become an essential tool for evidence-informed decision making [[4]Katrak P. Bialocerkowski A. Massy-Westropp N. Kumar V.S.S. Grimmer K. A systematic review of the content of critical appraisal tools.BMC Med Res Methodol. 2004; 4: 22Crossref PubMed Scopus (254) Google Scholar]. To summarize evidence, the Cochrane Collaboration group usually combines quantitative primary studies, especially randomized controlled trials. Other groups, such as the Campbell Collaboration, suggest that a systematic review should not be limited to quantitative studies and recognize that the inclusion of qualitative studies is often needed to answer some specific questions [[1]Popay J. Roberts H. Methodological issues in the synthesis of diverse sources of evidence.in: Popay J. Moving beyond effectiveness in evidence synthesis: methodological issues in the synthesis of diverse sources of evidence. National Institute for Health and Clinical Excellence, London, UK2006: 1-10Google Scholar]. A systematic review that concomitantly includes quantitative, qualitative, and mixed-methods primary studies is called mixed-methods review [[2]Pluye P. Gagnon M.-P. Griffiths F. Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in Mixed Studies Reviews.Int J Nurs Stud. 2009; 46: 529-546Abstract Full Text Full Text PDF PubMed Scopus (610) Google Scholar]. However, the integration of different study designs within the same review remains a challenge with regard to the assessment of methodological and reporting quality [[3]Dixon-Woods M. Bonas S. Booth A. Jones D.R. Miller T. Sutton A.J. et al.How can systematic reviews incorporate qualitative research? A critical perspective.Qual Res. 2006; 6: 27-44Crossref Scopus (476) Google Scholar]. Although several critical appraisal tools exist for systematic reviews, there is still no “gold standard” to date [[4]Katrak P. Bialocerkowski A. Massy-Westropp N. Kumar V.S.S. Grimmer K. A systematic review of the content of critical appraisal tools.BMC Med Res Methodol. 2004; 4: 22Crossref PubMed Scopus (254) Google Scholar]. Most available instruments were chiefly developed for the assessment of quantitative reviews and are not widely used, probably because of their length and complexity [4Katrak P. Bialocerkowski A. Massy-Westropp N. Kumar V.S.S. Grimmer K. A systematic review of the content of critical appraisal tools.BMC Med Res Methodol. 2004; 4: 22Crossref PubMed Scopus (254) Google Scholar, 5Crowe M. Sheppard L. A review of critical appraisal tools show they lack rigor: alternative tool structure is proposed.J Clin Epidemiol. 2011; 64: 79-89Abstract Full Text Full Text PDF PubMed Scopus (159) Google Scholar].In this letter, we would like to illustrate how the lack of consensus on reporting standards in mixed-methods reviews makes this type of reviews difficult to assess with the available tools. In 2009, we carried out a complex systematic overview on breast cancer risk perception and communication. In the context of this review, we compared the methodological quality of quantitative systematic reviews with mixed-methods reviews using two available assessment tools. The wide variety of reviews published on this topic provided an ideal context for this endeavor because the concepts of risk perception and communication in breast cancer have been studied extensively, using both quantitative and qualitative designs. The methodological quality of selected reviews was assessed independently by two reviewers using two critical appraisal tools: Overview of Quality Assessment Questionnaire (OQAQ) tool [[6]Oxman A.D. Guyatt G.H. Validation of an index of the quality of review articles.J Clin Epidemiol. 1991; 44: 1271-1278Abstract Full Text PDF PubMed Scopus (615) Google Scholar] and Assessment of Multiple Systematic Reviews (AMSTAR) tool [[7]Shea B. Grimshaw J. Wells G. Boers M. Andersson N. Hamel C. et al.Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews.BMC Med Res Methodol. 2007; 7: 10Crossref PubMed Scopus (2879) Google Scholar]. For both instruments, the score of each item was dichotomized: “yes” score vs. any other score. One point was attributed to each “yes” score, and the total was added up to calculate a final score. The final quality score for each review was determined based on a consensus score obtained after reviewing the items on which reviewers did not agree.The 22 selected reviews, published in peer-reviewed journals, include 12 systematic reviews of quantitative primary studies and 10 mixed-methods reviews (Table 1). The average AMSTAR score of quantitative reviews was significantly higher than that of mixed-methods reviews (P=0.005) (Table 2). With OQAQ, the average score of reviews including only quantitative studies also was higher than that of mixed-methods reviews (P=0.08) (Table 2). The differences observed between the two types of reviews can be explained by several factors. For instance, missing information about inclusion criteria (AMSTAR, item 4) and inappropriate use of the quality of included primary studies in formulating conclusions (AMSTAR, item 8) were the main weaknesses in the methodological and reporting aspects of mixed-methods reviews. Without detailed and adequate methodological reporting, it was difficult to assess the quality of review, independently of the assessment tool used.Table 1Characteristics of selected reviews and methodological quality scores (n=22)Author (year)JournalNumber of included studies (n)OQAQ consensus score (0–10)AMSTAR consensus score (0–11)Quantitative reviews (n=12) Albada (2009)Patient Educ Couns4098 Meilleur (2009)Am J Med Genet1354 Smerecnik (2009)J Gen Counsel1976 Sivell (2007)Cochrane398 Edwards (2006)Cochrane22910 Trevena (2006)J Eval Clin Pract4086 Braithwaite (2004)J Natl Cancer Inst2165 Butow (2003)Med J Aust1366 Meiser (2002)Soc Sci Med1278 Edwards (2001)J Health Commun2487 Rimer (1999)J Natl Cancer Inst2122 Vernon (1999)J Natl Cancer Inst2534Mixed-methods reviews (n=10) Visschers (2009)Risk Anal4554 Pilarski (2009)J Gen Counselna41 Edwards (2008)Patient Educ Couns2896 Sivell (2008)J Gen Counsel5976 Lipkus (2007)Med Decis Makingna12 Vahabi (2007)Health Educ9341 Epstein (2004)JAMA841 Walter (2004)Ann Fam Med1186 Julian-Reynier (2003)Eur J Hum Genet2313 Bottorff (1998)Patient Educ Couns7522Abbreviations: OQAQ, Overview of Quality Assessment Questionnaire; AMSTAR, Assessment of Multiple Systematic Reviews; na, not available. Open table in a new tab Table 2Component scores of OQAQ and AMSTAR for each type of reviewItemsQuantitative reviews (n=12)Mixed reviews (n = 10)Pn (%)n (%)OQAQ 1. Were the search methods used to find evidence (original research) on the primary question(s) stated?11 (92)7 (70)0.29 2. Was the search for evidence reasonably comprehensive?11 (92)7 (70)0.29 3. Were the criteria used for deciding which studies to include in the overview reported?10 (83)7 (70)0.29 4. Was bias in the selection of studies avoided?5 (42)1 (10)0.16 5. Were the criteria used for assessing the validity of the included studies reported?5 (42)2 (20)0.38 6. Was the validity of all studies referred to in the text assessed using appropriate criteria?7 (58)4 (40)0.67 7. Were the methods used to combine the findings of the relevant studies (to reach a conclusion) reported?8 (67)3 (30)0.39 8. Were the findings of the relevant studies combined appropriately relative to the primary question the overview addresses?10 (83)5 (50)0.17 9. Were the conclusions made by the author(s) supported by the data and/or analysis reported in the overview?12 (100)9 (90)0.45Mean score6.64.50.08AMSTAR 1. Was an “a priori” design provided?5 (42)4 (40)0.82 2. Was there duplicate study selection and data extraction?7 (58)4 (40)0.61 3. Was a comprehensive literature search performed?11 (92)8 (80)0.71 4. Was the status of publication (i.e., gray literature) used as an inclusion criterion?8 (67)1 (10)0.01 5. Was a list of studies (included and excluded) provided?4 (33)0 (0)0.10 6. Were the characteristics of the included studies provided?11 (92)5 (50)0.06 7. Was the scientific quality of the included studies assessed and documented?6 (50)2 (20)0.41 8. Was the scientific quality of the included studies used appropriately in formulating conclusions?11 (92)3 (30)0.01 9. Were the methods used to combine the findings of studies appropriate?3 (25)0 (0)0.39 10. Was the likelihood of publication bias assessed?0 (0)0 (0)0.65 11. Were potential conflicts of interest included?8 (70)5 (50)0.67Mean score6.23.20.005Abbreviations: OQAQ, Overview of Quality Assessment Questionnaire; AMSTAR, Assessment of Multiple Systematic Reviews. Open table in a new tab Although previous studies showed that OQAQ and AMSTAR are both reliable and valid to assess the quality of systematic reviews of randomized trials [6Oxman A.D. Guyatt G.H. Validation of an index of the quality of review articles.J Clin Epidemiol. 1991; 44: 1271-1278Abstract Full Text PDF PubMed Scopus (615) Google Scholar, 8Shea B. Hamel C. Wells G.A. Bouter L.M. Kristjansson E. Grimshaw J. et al.AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews.J Clin Epidemiol. 2009; 62: 1013-1020Abstract Full Text Full Text PDF PubMed Scopus (1255) Google Scholar], these tools have never been validated for mixed-methods reviews. The assessment of quality provided by these tools focuses on the reporting quality of the reviews. What is not reported is considered not to have been done by the authors. Therefore, the tools might be considered too unfair toward mixed-methods reviews. It is possible that mixed-methods reviews are published more frequently in journals that impose a maximum number of words to authors. This is a situation where authors tend to report less methodological aspects of their work and are more fluent on results and discussion sections [[9]Moja L.P. Telaro E. D’Amico R. Moschetti I. Coe L. Liberati A. et al.Assessment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study.BMJ. 2005; 330: 1053Crossref PubMed Scopus (168) Google Scholar]. Moja et al. [[9]Moja L.P. Telaro E. D’Amico R. Moschetti I. Coe L. Liberati A. et al.Assessment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study.BMJ. 2005; 330: 1053Crossref PubMed Scopus (168) Google Scholar] found that nearly 50% of published systematic reviews did not specify how the methodological quality of primary studies was evaluated or how it was considered in the interpretation of results, contrary to quantitative Cochrane reviews. Although systematic approaches have been proposed to achieve mixed-methods reviews [[1]Popay J. Roberts H. Methodological issues in the synthesis of diverse sources of evidence.in: Popay J. Moving beyond effectiveness in evidence synthesis: methodological issues in the synthesis of diverse sources of evidence. National Institute for Health and Clinical Excellence, London, UK2006: 1-10Google Scholar], at present days, there is no specific guideline. The realization of mixed-methods reviews is a methodological challenge in terms of study selection, assessment of quality of different types of primary studies, and synthesis of results [[2]Pluye P. Gagnon M.-P. Griffiths F. Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in Mixed Studies Reviews.Int J Nurs Stud. 2009; 46: 529-546Abstract Full Text Full Text PDF PubMed Scopus (610) Google Scholar]. Because the methodological aspects of qualitative studies and quantitative studies can be considerably different, it might be necessary to consider as many critical appraisal tools as there are types of systematic reviews. Our experience emphasizes the need for a consensus on standards for reporting quality of mixed-methods reviews. Without such a consensus, assessing this type of systematic review should be done with caution. Systematic reviews are central to knowledge translation [1Popay J. Roberts H. Methodological issues in the synthesis of diverse sources of evidence.in: Popay J. Moving beyond effectiveness in evidence synthesis: methodological issues in the synthesis of diverse sources of evidence. National Institute for Health and Clinical Excellence, London, UK2006: 1-10Google Scholar, 2Pluye P. Gagnon M.-P. Griffiths F. Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in Mixed Studies Reviews.Int J Nurs Stud. 2009; 46: 529-546Abstract Full Text Full Text PDF PubMed Scopus (610) Google Scholar, 3Dixon-Woods M. Bonas S. Booth A. Jones D.R. Miller T. Sutton A.J. et al.How can systematic reviews incorporate qualitative research? A critical perspective.Qual Res. 2006; 6: 27-44Crossref Scopus (476) Google Scholar] and have become an essential tool for evidence-informed decision making [[4]Katrak P. Bialocerkowski A. Massy-Westropp N. Kumar V.S.S. Grimmer K. A systematic review of the content of critical appraisal tools.BMC Med Res Methodol. 2004; 4: 22Crossref PubMed Scopus (254) Google Scholar]. To summarize evidence, the Cochrane Collaboration group usually combines quantitative primary studies, especially randomized controlled trials. Other groups, such as the Campbell Collaboration, suggest that a systematic review should not be limited to quantitative studies and recognize that the inclusion of qualitative studies is often needed to answer some specific questions [[1]Popay J. Roberts H. Methodological issues in the synthesis of diverse sources of evidence.in: Popay J. Moving beyond effectiveness in evidence synthesis: methodological issues in the synthesis of diverse sources of evidence. National Institute for Health and Clinical Excellence, London, UK2006: 1-10Google Scholar]. A systematic review that concomitantly includes quantitative, qualitative, and mixed-methods primary studies is called mixed-methods review [[2]Pluye P. Gagnon M.-P. Griffiths F. Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in Mixed Studies Reviews.Int J Nurs Stud. 2009; 46: 529-546Abstract Full Text Full Text PDF PubMed Scopus (610) Google Scholar]. However, the integration of different study designs within the same review remains a challenge with regard to the assessment of methodological and reporting quality [[3]Dixon-Woods M. Bonas S. Booth A. Jones D.R. Miller T. Sutton A.J. et al.How can systematic reviews incorporate qualitative research? A critical perspective.Qual Res. 2006; 6: 27-44Crossref Scopus (476) Google Scholar]. Although several critical appraisal tools exist for systematic reviews, there is still no “gold standard” to date [[4]Katrak P. Bialocerkowski A. Massy-Westropp N. Kumar V.S.S. Grimmer K. A systematic review of the content of critical appraisal tools.BMC Med Res Methodol. 2004; 4: 22Crossref PubMed Scopus (254) Google Scholar]. Most available instruments were chiefly developed for the assessment of quantitative reviews and are not widely used, probably because of their length and complexity [4Katrak P. Bialocerkowski A. Massy-Westropp N. Kumar V.S.S. Grimmer K. A systematic review of the content of critical appraisal tools.BMC Med Res Methodol. 2004; 4: 22Crossref PubMed Scopus (254) Google Scholar, 5Crowe M. Sheppard L. A review of critical appraisal tools show they lack rigor: alternative tool structure is proposed.J Clin Epidemiol. 2011; 64: 79-89Abstract Full Text Full Text PDF PubMed Scopus (159) Google Scholar]. In this letter, we would like to illustrate how the lack of consensus on reporting standards in mixed-methods reviews makes this type of reviews difficult to assess with the available tools. In 2009, we carried out a complex systematic overview on breast cancer risk perception and communication. In the context of this review, we compared the methodological quality of quantitative systematic reviews with mixed-methods reviews using two available assessment tools. The wide variety of reviews published on this topic provided an ideal context for this endeavor because the concepts of risk perception and communication in breast cancer have been studied extensively, using both quantitative and qualitative designs. The methodological quality of selected reviews was assessed independently by two reviewers using two critical appraisal tools: Overview of Quality Assessment Questionnaire (OQAQ) tool [[6]Oxman A.D. Guyatt G.H. Validation of an index of the quality of review articles.J Clin Epidemiol. 1991; 44: 1271-1278Abstract Full Text PDF PubMed Scopus (615) Google Scholar] and Assessment of Multiple Systematic Reviews (AMSTAR) tool [[7]Shea B. Grimshaw J. Wells G. Boers M. Andersson N. Hamel C. et al.Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews.BMC Med Res Methodol. 2007; 7: 10Crossref PubMed Scopus (2879) Google Scholar]. For both instruments, the score of each item was dichotomized: “yes” score vs. any other score. One point was attributed to each “yes” score, and the total was added up to calculate a final score. The final quality score for each review was determined based on a consensus score obtained after reviewing the items on which reviewers did not agree. The 22 selected reviews, published in peer-reviewed journals, include 12 systematic reviews of quantitative primary studies and 10 mixed-methods reviews (Table 1). The average AMSTAR score of quantitative reviews was significantly higher than that of mixed-methods reviews (P=0.005) (Table 2). With OQAQ, the average score of reviews including only quantitative studies also was higher than that of mixed-methods reviews (P=0.08) (Table 2). The differences observed between the two types of reviews can be explained by several factors. For instance, missing information about inclusion criteria (AMSTAR, item 4) and inappropriate use of the quality of included primary studies in formulating conclusions (AMSTAR, item 8) were the main weaknesses in the methodological and reporting aspects of mixed-methods reviews. Without detailed and adequate methodological reporting, it was difficult to assess the quality of review, independently of the assessment tool used. Abbreviations: OQAQ, Overview of Quality Assessment Questionnaire; AMSTAR, Assessment of Multiple Systematic Reviews; na, not available. Abbreviations: OQAQ, Overview of Quality Assessment Questionnaire; AMSTAR, Assessment of Multiple Systematic Reviews. Although previous studies showed that OQAQ and AMSTAR are both reliable and valid to assess the quality of systematic reviews of randomized trials [6Oxman A.D. Guyatt G.H. Validation of an index of the quality of review articles.J Clin Epidemiol. 1991; 44: 1271-1278Abstract Full Text PDF PubMed Scopus (615) Google Scholar, 8Shea B. Hamel C. Wells G.A. Bouter L.M. Kristjansson E. Grimshaw J. et al.AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews.J Clin Epidemiol. 2009; 62: 1013-1020Abstract Full Text Full Text PDF PubMed Scopus (1255) Google Scholar], these tools have never been validated for mixed-methods reviews. The assessment of quality provided by these tools focuses on the reporting quality of the reviews. What is not reported is considered not to have been done by the authors. Therefore, the tools might be considered too unfair toward mixed-methods reviews. It is possible that mixed-methods reviews are published more frequently in journals that impose a maximum number of words to authors. This is a situation where authors tend to report less methodological aspects of their work and are more fluent on results and discussion sections [[9]Moja L.P. Telaro E. D’Amico R. Moschetti I. Coe L. Liberati A. et al.Assessment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study.BMJ. 2005; 330: 1053Crossref PubMed Scopus (168) Google Scholar]. Moja et al. [[9]Moja L.P. Telaro E. D’Amico R. Moschetti I. Coe L. Liberati A. et al.Assessment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study.BMJ. 2005; 330: 1053Crossref PubMed Scopus (168) Google Scholar] found that nearly 50% of published systematic reviews did not specify how the methodological quality of primary studies was evaluated or how it was considered in the interpretation of results, contrary to quantitative Cochrane reviews. Although systematic approaches have been proposed to achieve mixed-methods reviews [[1]Popay J. Roberts H. Methodological issues in the synthesis of diverse sources of evidence.in: Popay J. Moving beyond effectiveness in evidence synthesis: methodological issues in the synthesis of diverse sources of evidence. National Institute for Health and Clinical Excellence, London, UK2006: 1-10Google Scholar], at present days, there is no specific guideline. The realization of mixed-methods reviews is a methodological challenge in terms of study selection, assessment of quality of different types of primary studies, and synthesis of results [[2]Pluye P. Gagnon M.-P. Griffiths F. Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in Mixed Studies Reviews.Int J Nurs Stud. 2009; 46: 529-546Abstract Full Text Full Text PDF PubMed Scopus (610) Google Scholar]. Because the methodological aspects of qualitative studies and quantitative studies can be considerably different, it might be necessary to consider as many critical appraisal tools as there are types of systematic reviews. Our experience emphasizes the need for a consensus on standards for reporting quality of mixed-methods reviews. Without such a consensus, assessing this type of systematic review should be done with caution.