Survivors of prostate, bladder, and colorectal cancer endure many sexual side-effects of treatment that negatively impact their relationships and diminish their quality of life. Multiple barriers exist in addressing men's sexual concerns in oncological care. To describe barriers of sexual recovery in men with prostate, bladder, and colorectal cancer. We searched PubMed for peer-reviewed, English-language articles published from 1999 to 2019 using the following search terms: "prostate cancer," or "bladder cancer," or "colorectal cancer," and "male," and "sexual function," or "sexual barrier" or "sexual dysfunction." Criteria for inclusion consisted of peer-reviewed articles (review, cross-sectional, longitudinal, interventional, or pilot studies) addressing sexual issues in men with a history of prostate, bladder, or colorectal cancer. Barriers to sexual recovery in men with prostate, bladder, and colorectal cancer include psychosocial barriers such as the feeling of loss, grief, depression and anxiety, the poor utilization, and excessive cost of pro-erectile aids, a diminished sense of masculinity and reluctance to seek help for sexual problems, as well as poor couple coping. Barriers in healthcare also exist, as healthcare providers often do not effectively address sexual issues due to poor communication, lack of comfort in discussing sexual issues, time constraints, and patients' hesitation to initiate discussions on sexual dysfunction. Patients with stomas and gay, bisexual, and queer men face additional challenges in their recovery of sexual intimacy. Barriers to sexual recovery are present in men during all stages of cancer and all modalities of treatment including surgery, radiation, or androgen deprivation therapy. There are multiple overlapping psychosocial and healthcare system barriers to sexual recovery after prostate, bladder, and colorectal cancer treatment. Oncological providers must be cognizant of these complex barriers so they can facilitate patients' access to resources needed for successful sexual recovery after genitourinary cancer treatment. Evidence based interventions, such as couple psychosexual counseling and peer support should be implemented via multidisciplinary care.