Independent external validation of QCancer (Ovarian)

Abstract

We read Collins and Altman's excellent validation of QCancer® (Ovarian) with particular interest (Collins & Altman 2012). While Collins and Altman reported that the algorithm was well calibrated across all tenths of risk, we note that they found some overprediction in the older age groups. In the original study (Hippisley-Cox & Coupland 2012), the outcome for ovarian cancer was defined as a diagnosis of ovarian cancer recorded either in the general practitioner (GP) record or in the linked cause of death record. As described in the paper (Hippisley-Cox & Coupland 2012), using both GP and Office of National Statistics death data sources for the definition of the outcome increased the ascertainment of cases by around 10–12% on the derivation and validation samples. The independent external validation reported by Collins and Altman is based on a separate database (THIN). The THIN database is not linked to cause of death data so the outcome for ovarian cancer in the Collins study is necessarily based solely on the data recorded in the GP record. This is therefore likely to result in a lower incidence rate of ovarian cancer in the THIN database compared with QResearch. Table 1 compares the incidence of ovarian cancer in the QResearch validation database using two definitions: (1) ovarian cancer recorded in either GP or deaths data as in the original study; and (2) ovarian cancer only recorded in the GP data. The table also shows the percentage of ovarian cancer cases identified only on the GP data in QResearch and how the ascertainment falls with increasing age. When a direct comparison of the incidence of ovarian cancer is made using identical case definitions, the rates are much closer. Since the incidence rate of the outcome will affect the predictive algorithm, we hypothesised therefore that the apparent overprediction of QCancer® when applied to THIN particularly at older ages is likely to reflect the under-ascertainment of ovarian cancer in the THIN database. Figure 1 shows the calibration of QCancer® by age group in the original QResearch validation cohort based on the original case definition. Figure 2 shows the calibration by age in QResearch when only diagnoses recorded on the GP record are included in the validation cohort. The graph shows a similar pattern to that in the Collins paper with apparent over prediction at older ages. We believe these results show that rather than QCancer® overpredicting, the discrepancy is mainly due to a lower incidence of the ovarian cancer in THIN since it does not include additional cases recorded on death data. Calibration of QCancer in QResearch based on the original definition of ovarian cancer, that is, recorded in either the general practitioner (GP) or Office of National Statistics (ONS) death record. Calibration of QCancer in QResearch based on the amended definition of ovarian cancer, that is, recorded solely in the general practitioner (GP) records. J.H.-C. is professor of clinical epidemiology at the University of Nottingham and co-director of QResearch® – a not-for-profit organisation, which is a joint partnership between the University of Nottingham and EMIS (leading commercial supplier of IT for 60% of general practices in the UK). J.H.-C. is also a paid director of ClinRisk Ltd, which produces software to ensure the reliable and updatable implementation of clinical risk algorithms within clinical computer systems to help improve patient care. C.C. is associate professor of Medical Statistics at the University of Nottingham and a paid consultant statistician for ClinRisk Ltd. This work and any views expressed within it are solely those of the co-authors and not of any affiliated bodies or organisations.