The retinoic acid receptor-related orphan receptor γ (RORγ) is regarded as an attractive therapeutic target for the treatment of prostate cancer. Herein, we report the identification, optimization, and evaluation of 1,2,3,4-tetrahydroquinoline derivatives as novel RORγ inverse agonists, starting from high throughput screening using a thermal stability shift assay (TSA). The representative compounds 13e (designated as XY039) and 14a (designated as XY077) effectively inhibited the RORγ transcriptional activity and exhibited excellent selectivity against other nuclear receptor subtypes. The structural basis for their inhibitory potency was elucidated through the crystallographic study of RORγ LBD complex with 13e. Both 13e and 14a demonstrated reasonable antiproliferative activity, potently inhibited colony formation and the expression of AR, AR regulated genes, and other oncogene in AR positive prostate cancer cell lines. Moreover, 13e and 14a effectively suppressed tumor growth in a 22Rv1 xenograft tumor model in mice. This work provides new and valuable lead compounds for further development of drugs against prostate cancer.