We evaluated the prognosis of the new grade groups and American Joint Committee on Cancer (AJCC) stage groups in men with prostate cancer (PCa) who were treated conservatively. A total of 13 798 eligible men were chosen from the Surveillance Epidemiology and End Results database. The new grade and AJCC stage groups were investigated on prostate biopsy specimens. Kaplan–Meier survival analysis and multivariable hazards models were applied to estimate the association of new grade and stage groups with overall survival (OS) and PCa-specific survival (CSS). Mean follow-up was 42.65 months (95% confidence interval: 42.47–42.84) in the entire cohort. The 3-year OS and CSS rates stepped down for grade groups 1–5 and AJCC stage groups I–IVB, respectively. After adjusting for clinical and pathological characteristics, all grade groups and AJCC stage groups were associated with higher all-cause and PCa-specific mortality compared to the reference group (all P ≤ 0.003). In conclusion, we evaluated the oncological outcome of the new grade and AJCC stage groups on biopsy specimens of conservatively treated PCa. These two novel clinically relevant classifications can assist physicians to determine different therapeutic strategies for PCa patients.