Abstract B204: A DREAM Challenge to improve prognostic models in patients with metastatic castrate-resistant prostate cancer

Abstract

Abstract Background: Crowdsourced competitions or “challenges” have proven effective at drawing large cross-disciplinary teams of experts to solve complex problems. However, no challenges to date have been conducted using cancer clinical trial data due to privacy, legal concerns, and the restricted availability of data. Project Data Sphere, LLC (PDS) and Sage Bionetworks/DREAM have recently completed the “Prostate Cancer DREAM Challenge” (Challenge) using historical comparator arm phase III clinical trial data from PDS. The goals of this Challenge were to advance understanding of disease progression and treatment toxicity in order to improve clinical management for patients with metastatic, castrate resistant prostate cancer (mCRPC). Method: Clinical variables, outcomes and lab test data from the control arms of four phase III clinical trial data sets in mCRPC - over 2,000 patients - were curated by PDS to create training and validation data sets for the Challenge. A team of prostate cancer clinicians and scientists selected 2 Challenge questions: prediction of overall survival (OS), and prediction of treatment discontinuation from standard-of-care (docetaxel). The Challenge opened in March, 2015, lasted over 4 months, and included multiple leaderboard rounds with a final validation round to assess overall winners. Subchallenge 1 scored participants' models using concordance index and an integrated area under receiver operator curve (iAUC) from 0-24 months; Subchallenge 2 scored models using area under precision-recall curve (AUPRC). A recently published mCRPC OS model (Halabi, et al., JCO, 2014) was used as the baseline benchmark for Subchallenge 1. All Challenge participants were required to provide complete documentation of their methods including reproducible code. Challenge details can be found at: www.synapse.org/#!Synapse:syn2813558. Results: The Challenge had over 160 active participants submitting a total of 852 models for leaderboard scoring, with 50 and 30 teams submitting final models for Subchallenges 1 and 2, respectively. Median iAUC for Subchallenge 1 was 0.76 (0.67-0.78) with a maximum score of 0.792 (scores > 0.5 are better than random). Over half (n = 35) of these models exceeded the existing benchmark (.743 iAUC). For Subchallenge 2, median AUPRC was 0.13 (0.09-0.17) with a maximum score of 0.19 (scores > 0.13 are better than random). Successful teams utilized techniques for careful imputation of missing data, and ensemble methods for aggregation of multiple machine learning models. Conclusion: The Prostate Cancer DREAM Challenge successfully engaged a diverse community of experts focusing on two clinically relevant questions in mCRPC. New models for predicting OS were proposed and validated with significant improvements over a prior benchmark. Also, clinical factors related to prediction of treatment discontinuation due to adverse events were uncovered, thereby establishing a novel benchmark and providing a strong rationale for follow-up studies. Importantly, this Challenge demonstrates the benefits of crowdsourcing important biomedical questions in cancer using clinical trial data. Citation Format: Justin Guinney, Tao Wang, Chris Bare, Thea Norman, Brian Bot, Liji Shen, Kimberly Winner, Stephen Friend, Kald Abdallah, Gustavo Stolovitzky, Yang Xie, Fang Liz Zhou, James C. Costello. A DREAM Challenge to improve prognostic models in patients with metastatic castrate-resistant prostate cancer. [abstract]. In: Proceedings of the AACR-NCI-EORTC International Conference: Molecular Targets and Cancer Therapeutics; 2015 Nov 5-9; Boston, MA. Philadelphia (PA): AACR; Mol Cancer Ther 2015;14(12 Suppl 2):Abstract nr B204.