To determine which factors predict for worsening erectile function after highly conformal, modestly hypofractionated radiotherapy to the prostate. All patients who received 60Gy in 20 fractions, volumetric modulated arc therapy to the prostate across 4 centers over 9 years were included in this study. The provincial electronic medical record was interrogated to identify any new prescriptions for erectile dysfunction (ED) medication, any change in prescription of ED medication or any permanent discontinuance of ED medication persisting beyond 6 months post completion of any androgen deprivation therapy. The penile bulb, penile crux and penile shaft structures were retrospectively contoured. A Youden receiver-operator-curve analysis, logistic regression, and neural network based interpretable machine learning analysis were then used to determine dependencies between worsening ED and clinical factors including mean doses to these structures. Two-hundred-twelve patients with median (inter-quartile-range) follow-up of 3.6 (3.2-4.4) years were identified. Median age was 72 (67-76) years. 104 (49%) patients received androgen deprivation therapy. Prior to treatment, 52 (25%) patients were on ED medication: 20 (9%) on sildenafil, 28 (13%) on tadalafil and 4 (8%) on vardenafil. Median PTV volume was 158.9 (129.8-192.1) cc. Median penile bulb, penile crux and penile shaft volumes were 4.7 (3.6-6.2) cc, 6.5 (5.1-8.5) cc and 93.3 (80.6-106.2) cc, respectively. PTV V95 was 99.8 (99.5-99.9)%. Mean doses to penile bulb, penile crux and penile shaft were 2094.8 (1306.2-3036.3) cGy, 2094.8 (1306.2-3036.3) cGy and 444.4 (313.2-650.5), respectively. Fifty-nine (28%) patients had a worsening of ED after treatment: 25 (12%) started a new ED medication, 6 (3%) had a prescription change and 28 (13%) stopped ED medication. On univariate analyses pretreatment use of ED medication predicted for worsening ED: odds ratio (OR) yes vs no: 10.2 (5.0 - 20.8; p<0.001). A trend towards mean dose to penile bulb [OR ≤2343.9 vs >2343.9: 1.7 (0.9-3.2; p = 0.08)] predicting for worsening ED was observed. Mean doses to penile crux [OR <1725.8 vs > 1725.8: 2.6 (1.3-5.2; p = 0.005)] and penile shaft [OR ≤344.9 vs >344.9: 5.2 (2.2-12.2; p<0.001)] predicted for worsening ED. Use of androgen deprivation therapy, and age at time of radiotherapy were not predictive of worsening ED. On multivariate analysis, only mean dose to penile shaft [OR ≤344.9 vs >344.9: 6.3 (1.9-20.3; p = 0.002)] and pretreatment use of ED medication [OR yes vs no: 11.1 (5.3-23.2; p<0.001)] predicted for worsening ED. A neural network analysis suggested that penile shaft mean dose and pre-treatment ED medication use are the most important factors in predicting worsening ED. In this limited analysis, pre-treatment use of ED medication and mean dose to penile shaft predicted for worsening ED after treatment with modestly hypofractionated radiotherapy for prostate cancer.