Given the increasing number of treatment options for patients with localized prostate cancer (PCa), there is a need for biomarkers to aid in risk stratification. Specifically, novel biomarkers can aid in the identification of high-risk phenotypes among similar patients in traditional NCCN risk groupings. One promising area for development is using pre-treatment prostate MRI to identify imaging biomarkers to identify prostate cancer patients at highest risk for recurrence. We hypothesized that deep learning could be leveraged to identify imaging biomarkers of aggressive PCa from pre-treatment prostate MRIs. Our study included 1,020 patients treated at our institution between 2010-2022. Given pathologic extraprostatic extension (EPE) and seminal vesicle invasion (SVI) are associated with higher risk of treatment failure, we hypothesized that deep learning models which identified radiographic EPE and SVI would provide non-invasive imaging biomarkers associated with PCa prognosis. We trained two separate deep learning models using convolutional neural networks to predict SVI and EPE respectively. The model inputs were T2W prostate MRIs (n = 894) and models consisted of 8 convolutional layers. Dropout, L2 regularization, and data augmentation were used to improve model generalizability and reduce overfitting. Discriminatory ability of each model was measured using AUC on a blinded external test set of 221 patients. To assess the clinical utility of our imaging biomarkers, log-rank tests were used to evaluate biochemical free survival (BFS) for patients classified as high risk to patients classified as low risk. Biochemical failure was defined as post-treatment PSA >0.1 for patients who underwent radical prostatectomy (RP) or PSA >2ng/ml above nadir for patients receiving radiation therapy. Within our cohort of 1,020 patients the median age was 66 with a median follow up of 4 years. 49.3% (n = 503) underwent RP and 50.7% (n = 517) received EBRT. 4% (n = 41) were low risk, 62.4% (n = 636) were intermediate risk, and 33% (n = 337) were high risk based on NCCN criteria. Deep learning models showed good discriminatory ability for both EPE (AUC 0.66) and SVI (AUC 0.74). Both imaging biomarkers showed prognostic ability to identify high risk prostate phenotypes. Patients deemed high risk based on EPE classifier had worse BFS (median 5 vs 9 years, p<.001). Similarly, patients classified as high risk based on SVI also showed worse BFS (median 5 vs 9 years, p = 0.024). Among intermediate risk patients, EPE biomarker showed an ability to identify high risk phenotypes (median 6 vs 9 years, p = 0.024). Deep learning classifiers of prostate MRIs demonstrated the ability to stratify high-risk prostate cancer phenotypes beyond traditional risk paradigms. Imaging biomarkers represent a non-invasive method to help aid in the personalization of treatment for patients with localized prostate cancer and identify patients who potentially require treatment escalation.