Background/Objectives: Patients with non-metastatic castration-resistant prostate cancer (nmCRPC) and high-risk features frequently have progression to life-threatening metastasis without second-generation antiandrogens. This study investigated nmCRPC patients for the survival and prognostic factors from a cohort before the approved use of second-generation antiandrogens. Methods: From March 2016 to January 2021, 326 patients treated with second-generation antiandrogens for metastatic castration-resistant prostate cancer (mCRPC) or metastatic castration-sensitive prostate cancer were retrieved. Forty-four patients experiencing nmCRPC with no use of second-generation antiandrogens were reviewed. The prognostic factors, at initial diagnosis or at nmCRPC, associated with metastasis-free survival (MFS) and overall survival (OS) were analyzed. Results: The median follow-up time after nmCRPC was 46 months. The median PSA level at nmCRPC was 2.7 ng/mL. Thirty-eight of forty-four patients with nmCRPC had a PSA doubling time (PSADT) of 10 months or shorter, and the median PSADT was 4 months. The median OS from nmCRPC was 53 months, and the median interval for nmCRPC patients progressing to mCRPC was 20 months. Upon univariate analysis, PSADT < 10 months (p = 0.049) and the very-high-risk group at the initial diagnosis (p = 0.043) were associated with significantly shorter post-nmCRPC MFS. The very-high-risk group (p = 0.031) was associated with significantly worse post-nmCRPC OS. In terms of survivals from the initial diagnosis of prostate cancer, Gleason grade ≥ 8 was the only independent factor with MFS and OS. Conclusions: Without second-generation antiandrogens, nmCRPC patients with PSADT <10 months and in the initial very-high-risk group developed subsequent mCRPC in a significantly faster fashion. Patients of the very-high-risk group had shorter survival rates after nmCRPC.