Improving Patient-Centered Outcomes through Methodological Rigor

Abstract

Although the medical community is highly dependent on published medical literature for assessing the accuracy and utility of molecular diagnostics, individual publications vary widely in the degree to which they include critical features of study design such as nature of the study population, the choice of reference standards, choice of statistical methods, and reporting of results. The use of differing design features such as these often is entirely appropriate, since studies that vary in these choices can inform medical professionals who perform these tests in a variety of clinical contexts.The Patient Centered Outcomes Research Institute (PCORI) recently published a draft methodology report1Gabriel S, Normand S-L, Aronson N, Basch E, Berg A, Flum D, Goodman S, Helfand M, Ioannidis J, Lauer M, Meltzer D, Mittman B, Newhouse R, Schneeweiss S, Slutsky J, Tinetti M, Yancy C: Draft methodology report: our questions, our decisions: standards for patient centered outcomes research. Public comment draft report. Patient-Centered Outcomes Research Institute (PCORI) Methodology Committee, July 23, 2012Google Scholar that suggests that studies of diagnostic tests should clearly specify clinical context and key elements of study design (which should be informed by the clinical context of testing), assess the effect of factors known to affect diagnostic performance and outcomes (such as threshold for reporting a positive result), give preference to prospective randomized designs (as opposed to observational studies), use appropriate statistical techniques to address confounding, and use structured reporting of diagnostic study results. Universal implementation of these recommendations in the short term is probably unrealistic, however. Randomized clinical trials are extremely expensive to undertake, and complete understanding of factors that may confound the interpretation of a diagnostic test would seem to require omniscience. Nevertheless, it is possible for The Journal of Molecular Diagnostics to advance the PCORI goals by identification of individual papers that clearly meet the criteria set forth by the Standards for Reporting of Diagnostic Accuracy (STARD) initiative.2Bossuyt P.M. Reitsma J.B. Bruns D.E. Gatsonis C.A. Glasziou P.P. Irwig L.M. Moher D. Rennie D. de Vet H.C. Lijmer J.G. The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration.Clin Chem. 2003; 49: 7-18Crossref PubMed Scopus (804) Google Scholar, 3Bossuyt P.M. Reitsma J.B. Bruns D.E. Gatsonis C.A. Glasziou P.P. Irwig L.M. Lijmer J.G. Moher D. Rennie D. de Vet H.C. Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative.Standards for Reporting of Diagnostic Accuracy Clin Chem. 2003; 49: 1-6Google Scholar, 4Bruns D.E. The STARD initiative and the reporting of studies of diagnostic accuracy.Clin Chem. 2003; 49: 19-20Crossref PubMed Scopus (39) Google Scholar These criteria have been enumerated in a checklist that addresses each element of a scientific paper and facilitates the description of key elements such as study aims, characteristics of human subjects and specimens, test methods, statistical evaluation, and results reporting. Further advancement of the PCORI goals can be achieved by increasing the availability of data and computer code used in attaining the reported results.Beginning in 2013, the JMD will begin to annotate papers that meet the STARD criteria with a footnote on the title page. Authors who wish their papers to receive this designation may submit a checklist that will form the basis for evaluation by the editorial staff at the time of acceptance. Participation is entirely voluntary, and the decision to participate will not directly affect the process by which we evaluate submitted papers. Authors should note, however, that some of the STARD criteria (such as calculation of 95% confidence intervals) are, and will remain, criteria for scientific evaluation. The Journal hopes that by adopting the STARD criteria in this way we can improve the communication between authors and readers.Baggerly and Coombes5Baggerly K.A. Coombes K.R. What information should be required to support clinical “omics” publications?.Clin Chem. 2011; 57: 688-690Crossref PubMed Scopus (47) Google Scholar have observed that the lack of transparency in omics studies has resulted in the initiation of clinical trials based on faulty conclusions and possible patient endangerment. They suggested that risk would be ameliorated were raw data and computer code publically available. Although the JMD has considered requiring the deposit of computer code and data as a prerequisite to publication, we believe that such a requirement, were it widely adopted, would discourage authors with a legitimate proprietary interest from submitting their omics studies to high quality peer-reviewed journals. For this reason, also beginning in 2013, we will allow authors to annotate papers for which they agree to make data and code available to qualified investigators for a minimum of five years following publication. Although the Journal encourages authors to make these materials available in a public form, as described in the Instructions for Authors, we realize that in some cases access should result from a formal agreement, such as a Data Use Agreement. Once again, the decision to annotate an article to indicate the intention of authors to make raw data and code available to readers is voluntary.Although participation in the annotation programs is voluntary, reviewers and editors may decide that for an individual paper, one or more of the elements is necessary for the paper to be publishable. The creation of these opportunities for voluntary annotation does not represent a change in editorial policy and will not be used to restrict reviewer discretion in the evaluation of submissions. The Journal of Molecular Diagnostics encourages authors to participate in these annotation programs, which reflect the strong commitment of the Journal’s Editors, the Editorial Board, and Editorial Office staff to continual improvement of the Journal’s scientific quality and utility to its readers. Although the medical community is highly dependent on published medical literature for assessing the accuracy and utility of molecular diagnostics, individual publications vary widely in the degree to which they include critical features of study design such as nature of the study population, the choice of reference standards, choice of statistical methods, and reporting of results. The use of differing design features such as these often is entirely appropriate, since studies that vary in these choices can inform medical professionals who perform these tests in a variety of clinical contexts. The Patient Centered Outcomes Research Institute (PCORI) recently published a draft methodology report1Gabriel S, Normand S-L, Aronson N, Basch E, Berg A, Flum D, Goodman S, Helfand M, Ioannidis J, Lauer M, Meltzer D, Mittman B, Newhouse R, Schneeweiss S, Slutsky J, Tinetti M, Yancy C: Draft methodology report: our questions, our decisions: standards for patient centered outcomes research. Public comment draft report. Patient-Centered Outcomes Research Institute (PCORI) Methodology Committee, July 23, 2012Google Scholar that suggests that studies of diagnostic tests should clearly specify clinical context and key elements of study design (which should be informed by the clinical context of testing), assess the effect of factors known to affect diagnostic performance and outcomes (such as threshold for reporting a positive result), give preference to prospective randomized designs (as opposed to observational studies), use appropriate statistical techniques to address confounding, and use structured reporting of diagnostic study results. Universal implementation of these recommendations in the short term is probably unrealistic, however. Randomized clinical trials are extremely expensive to undertake, and complete understanding of factors that may confound the interpretation of a diagnostic test would seem to require omniscience. Nevertheless, it is possible for The Journal of Molecular Diagnostics to advance the PCORI goals by identification of individual papers that clearly meet the criteria set forth by the Standards for Reporting of Diagnostic Accuracy (STARD) initiative.2Bossuyt P.M. Reitsma J.B. Bruns D.E. Gatsonis C.A. Glasziou P.P. Irwig L.M. Moher D. Rennie D. de Vet H.C. Lijmer J.G. The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration.Clin Chem. 2003; 49: 7-18Crossref PubMed Scopus (804) Google Scholar, 3Bossuyt P.M. Reitsma J.B. Bruns D.E. Gatsonis C.A. Glasziou P.P. Irwig L.M. Lijmer J.G. Moher D. Rennie D. de Vet H.C. Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative.Standards for Reporting of Diagnostic Accuracy Clin Chem. 2003; 49: 1-6Google Scholar, 4Bruns D.E. The STARD initiative and the reporting of studies of diagnostic accuracy.Clin Chem. 2003; 49: 19-20Crossref PubMed Scopus (39) Google Scholar These criteria have been enumerated in a checklist that addresses each element of a scientific paper and facilitates the description of key elements such as study aims, characteristics of human subjects and specimens, test methods, statistical evaluation, and results reporting. Further advancement of the PCORI goals can be achieved by increasing the availability of data and computer code used in attaining the reported results. Beginning in 2013, the JMD will begin to annotate papers that meet the STARD criteria with a footnote on the title page. Authors who wish their papers to receive this designation may submit a checklist that will form the basis for evaluation by the editorial staff at the time of acceptance. Participation is entirely voluntary, and the decision to participate will not directly affect the process by which we evaluate submitted papers. Authors should note, however, that some of the STARD criteria (such as calculation of 95% confidence intervals) are, and will remain, criteria for scientific evaluation. The Journal hopes that by adopting the STARD criteria in this way we can improve the communication between authors and readers. Baggerly and Coombes5Baggerly K.A. Coombes K.R. What information should be required to support clinical “omics” publications?.Clin Chem. 2011; 57: 688-690Crossref PubMed Scopus (47) Google Scholar have observed that the lack of transparency in omics studies has resulted in the initiation of clinical trials based on faulty conclusions and possible patient endangerment. They suggested that risk would be ameliorated were raw data and computer code publically available. Although the JMD has considered requiring the deposit of computer code and data as a prerequisite to publication, we believe that such a requirement, were it widely adopted, would discourage authors with a legitimate proprietary interest from submitting their omics studies to high quality peer-reviewed journals. For this reason, also beginning in 2013, we will allow authors to annotate papers for which they agree to make data and code available to qualified investigators for a minimum of five years following publication. Although the Journal encourages authors to make these materials available in a public form, as described in the Instructions for Authors, we realize that in some cases access should result from a formal agreement, such as a Data Use Agreement. Once again, the decision to annotate an article to indicate the intention of authors to make raw data and code available to readers is voluntary. Although participation in the annotation programs is voluntary, reviewers and editors may decide that for an individual paper, one or more of the elements is necessary for the paper to be publishable. The creation of these opportunities for voluntary annotation does not represent a change in editorial policy and will not be used to restrict reviewer discretion in the evaluation of submissions. The Journal of Molecular Diagnostics encourages authors to participate in these annotation programs, which reflect the strong commitment of the Journal’s Editors, the Editorial Board, and Editorial Office staff to continual improvement of the Journal’s scientific quality and utility to its readers.