Abstract P2-03-09: Benchmarking mutation function prediction algorithms using validated cancer driver and passenger mutations

Abstract

Abstract Background: Massively parallel sequencing studies have identified large numbers of mutations of unknown biologic significance. There is a pressing need for computational methods to predict and distinguish neutral from potentially pathogenic mutations accurately, to help identify those mutations worth exploring experimentally and clinically. Although various bioinformatic algorithms are available, they are based on different methodologies and assumptions, and their predictions for the same mutations are not always concordant. In this study, we sought to benchmark the performance of 17 prediction algorithms using functionally validated and pathognomonic mutations. Methods: We curated the literature for functionally validated and pathognomonic mutations as our positive dataset (i.e. pathogenic mutations). For the negative dataset (i.e. neutral mutations), we retrieved variants from the dbSNP database, including only those with minor allele frequency >25%. We compiled a total of 7975 mutations (875 pathogenic and 7100 neutral). The performance of each prediction algorithm, namely accuracy, specificity, sensitivity, positive predictive value (PPV) and negative predictive value (NPV), were defined using the positive and negative datasets described above. Confidence intervals were calculated by sub-sampling 2/3 of the functionally pathogenic mutations and equal number of neutral mutations 500 times. To reduce the bias introduced by mutations included in the COSMIC database, we excluded those found in COSMIC v67, resulting in 6048 mutations (212 pathogenic and 5835 neutral), and re-evaluated the performance of each prediction algorithm. Results: Our analysis revealed that the overall accuracy varied considerably, with a median of 87% (range 78%-97%). In terms of accuracy, FATHMM (cancer) statistically outperformed all other prediction algorithms (97%, 95% confidence interval (CI) 96%-98%), followed by MutationTaster 2 (94%, 95% CI 93%-95%). Sensitivity and specificity also varied (median 85%, range 77%-96% and median 89%, range 71%-100%, respectively). The most sensitive prediction algorithm, FATHMM (cancer) (96%, 95% CI 95%-97%) statistically outperformed all others. The most specific prediction algorithm was CHASM (breast) (100%, 95% CI 94%-100%). While CHASM (breast) had the highest PPV (100%, 95% CI 99%-100%), FATHMM (cancer) had statistically better NPV than all other prediction algorithms (96%, 95% CI 95%-97%). When COSMIC mutations were removed, FATHMM (cancer) remained the most accurate (93%, 95% CI 91%-95%) though the difference was not statistically significant. In this context, CanDrA (breast) was the most sensitive prediction algorithm (95%, 95% CI 93%-97%) and had the highest NPV (93%, 95% CI 90%-96%), while CHASM (breast) was the most specific prediction algorithm (100%, 95% CI 99%-100%) and had the best PPV (99%, 95% CI 97%-100%). Conclusions: Our results demonstrate that functional prediction algorithms varied in performance. Using this dataset of mutations, FATHMM (cancer) outperformed all other prediction algorithms in terms of accuracy, sensitivity and NPV, and remained the most accurate even when mutations catalogued in the COSMIC database were excluded. Citation Format: Maria R De Filippo, Charlotte KY Ng, Jorge S Reis-Filho, Britta Weigelt. Benchmarking mutation function prediction algorithms using validated cancer driver and passenger mutations [abstract]. In: Proceedings of the Thirty-Seventh Annual CTRC-AACR San Antonio Breast Cancer Symposium: 2014 Dec 9-13; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2015;75(9 Suppl):Abstract nr P2-03-09.