Roughly 30% of men with prostate cancer who undergo radical prostatectomy will suffer biochemical cancer recurrence (BCR). Accurately predicting which patients will experience BCR could identify who would benefit from increased surveillance or adjuvant therapy. Unfortunately, no current method can effectively predict this. We develop and evaluate PathCLR, a novel semi-supervised method that learns a model that can use hematoxylin and eosin (H&E)-stained tissue microarrays (TMAs) to predict prostate cancer recurrence within 5 years after diagnosis. The learning process involves 2 sequential steps: PathCLR (a) first employs self-supervised learning to generate effective feature representations of the input images, then (b) feeds these learned features into a fully supervised neural network classifier to learn a model for predicting BCR. We conducted training and evaluation using 2 large prostate cancer datasets: (1) the Cooperative Prostate Cancer Tissue Resource (CPCTR) with 374 patients, including 189 who experienced BCR, and (2) the Johns Hopkins University (JHU) prostate cancer dataset of 646 patients, with 451 patients having BCR. PathCLR’s (10-fold cross-validation) F1 score was 0.61 for CPCTR and 0.85 for JHU. This was statistically superior (paired t-test with P <.05) to the best-learned model that relied solely on clinicopathological features, including PSA level, primary and secondary Gleason Grade, etc. We attribute the improvement of PathCLR over models using only clinicopathological features to its utilization of both learned latent representations of tissue core images and clinicopathological features. This finding suggests that there is essential predictive information in tissue images at the time of surgery that goes beyond the knowledge obtained from reported clinicopathological features, helping predict the patient’s 5-year outcome.