Sexual dysfunction, including erectile dysfunction and loss of libido, are common among men undergoing treatment for localized prostate cancer. Both local treatments and systemic androgen deprivation therapy may contribute to these outcomes and are differentially indicated based on disease characteristics. We sought to compare sexual function through 5 years after radiation treatment with and without androgen deprivation therapy in men with good baseline sexual function to better understand long-term effects in this understudied subset of patients. We retrospectively reviewed a prospectively assembled population-based cohort of men who underwent radiation with and without androgen deprivation therapy for intermediate or high-risk localized prostate cancer. Sexual function was assessed longitudinally over 5 years. Men with erections sufficient for intercourse at baseline were selected for inclusion. Out of 167 patients included, 73 underwent radiation alone and 94 received androgen deprivation therapy plus radiation (51 with intermediate and 43 with high-risk disease). Androgen deprivation therapy use was associated with worse sexual function through 1 year regardless of disease risk. This difference was no longer statistically significant at 3 years in the intermediate-risk group. Compared to radiation alone, androgen deprivation therapy in high-risk disease was associated with worse sexual function at 3 years (effect: -20.3 points, CI [-31.8, -8.8], p < 0.001) but not at 5 years (effect: -3.4, CI [-17.2, 10.5], p = 0.63). Androgen deprivation therapy plus radiation is associated with worse sexual function through 3-years follow-up in men with high-risk prostate cancer compared to radiation alone. The addition of androgen deprivation therapy in the treatment of intermediate-risk disease does not appear to result in worse sexual function at 3 or 5-year follow-up compared to radiation alone.