We estimated the risks of prostate cancer-specific mortality (PCSM) and all-cause mortality (ACM) in men with high-risk prostate cancer (PC) undergoing external beam radiation therapy and brachytherapy with short-course androgen deprivation therapy (ADT) (median 4 months) as compared with men with more favorable-risk PC undergoing standard of care as per the National Comprehensive Cancer Network guidelines. The prospective study cohort comprised 6595 consecutively treated men with T1-4 N0M0 PC whose treatment included brachytherapy between October 16, 1997, and May 28, 2013. Fine and Gray competing risk regression and Cox regression analyses were used to assess the risks of PCSM and ACM in men with high, unfavorable intermediate, and favorable intermediate risk as compared with low-risk PC. After median followup of 7.76 years, 820 men died (12.43%): 72 of PC (8.78%). Men with favorable intermediate-risk PC did not have significantly increased PCSM risk as compared with men with low-risk PC (adjusted hazard ratio [AHR], 1.26; 95% confidence interval [CI] 0.56, 2.88; p-Value 0.58), whereas men with high-risk PC (AHR, 3.74; 95% CI 1.12, 12.53; p-Value 0.032) and unfavorable intermediate-risk PC (AHR, 3.10; 95% CI 1.43, 6.72; p-Value 0.004) did. Based on 10-year adjusted point estimates of PCSM and ACM for men with high-risk PC being 6.01% (95% CI 3.79%, 8.94%) and 21.30% (95% CI 17.45%, 25.42%), respectively, PCSM comprised 28% of ACM. In the setting of external beam radiation therapy and brachytherapy, men with high-risk PC have low absolute adjusted estimates of PCSM (~6%) during the first decade after treatment despite receiving only short-course ADT. Whether long-term ADT can lower PCSM and improve survival in these men requires additional study.