Prediction of Outcome of Assisted Reproductive Technologies (ART): Is it Reported With Traditional Parameters of Accuracy or Calibration?

Abstract

Prediction of the outcome of assisted reproductive technologies (ART) is an important issue in reproductive medicine. The capacity of a test to predict outcome can be expressed in terms of its accuracy (as in sensitivity, specificity) or in terms of its calibration. Calibration expresses the agreement between a probability as predicted by a test and the true success rate as observed in clinical practice. Although it is known for more than a decade that the calibration of a test is as important as its discriminative capacity, it is questionable whether this principle is applied in the assessment of diagnostic and prognostic tests in reproductive medicine. The aim of this study was to study how prediction of outcome of assisted reproductive technologies with a prognostic test is reported in two fertility journals. Quantitative literature search. We searched all issues of Fertility and Sterility and Human Reproduction published in 1999 and 2004 for articles on the prediction of ART outcome. Articles about randomized trials, meta-analyses, guidelines and non-systematic reviews as well as letters-to-the-editors and editorials were not considered. The remaining articles were selected for further reading based on their title and abstract. We included studies when they reported on a prognostic test or a prediction model in ART (IUI or IVF) for clinical pregnancy. For each article, we recorded how predictive capacity was expressed, whether data on calibration were provided, and what conclusions were drawn by the authors. A total number of 1365 articles were identified of which 37 articles were selected on the basis of their abstract, and 32 articles were included after reading the whole article. Of these, 14 (44%) reported on test accuracy, expressed in terms of sensitivity and specificity (n=11, 34%), an ROC-curve in 9 studies (28%), or as predictive values in 7 studies (22%). In another four articles it was possible to reconstruct 2x2 tables. Statistical uncertainty was assessed with 95%-confidential intervals in 3 articles (9.4%), whereas 95% confidence intervals could be obtained from the data provided in another 15 articles. Data on calibration were reported in only 3 studies (9%). Conclusions about whether or not the test should be used in daily practice were made in 27 studies (84%). These conclusions were supported by data on test accuracy in 63% of these studies, and by data on calibration in 11% of these studies. Prediction of the outcome of ART, if reported, has mainly been summarized with indices of diagnostic test accuracy, and statistical uncertainty is rarely quantified. Only a very few studies report on the calibration of a test. Recommendations for daily practice are often made without support of data provided in clinical studies. Editors of medical journals in reproductive medicine should reconsider their review policy of studies on prognostic tests.