Prostate cancer (PCa) is an extensive heterogeneous disease with a complex cellular ecosystem in the tumor microenvironment (TME). However, the manner in which heterogeneity is shaped by tumors and stromal cells, or vice versa, remains poorly understood. In this study, single-cell RNA sequencing, spatial transcriptomics, and bulk ATAC-sequence are integrated from a series of patients with PCa and healthy controls. A stemness subset of club cells marked with SOX9highARlow expression is identified, which is markedly enriched after neoadjuvant androgen-deprivation therapy (ADT). Furthermore, a subset of CD8+CXCR6+ T cells that function as effector T cells is markedly reduced in patients with malignant PCa. For spatial transcriptome analysis, machine learning and computational intelligence are comprehensively utilized to identify the cellular diversity of prostate cancer cells and cell-cell communication in situ. Macrophage and neutrophil state transitions along the trajectory of cancer progression are also examined. Finally, the immunosuppressive microenvironment in advanced PCa is found to be associated with the infiltration of regulatory T cells (Tregs), potentially induced by an FAP+ fibroblast subset. In summary, the cellular heterogeneity is delineated in the stage-specific PCa microenvironment at single-cell resolution, uncovering their reciprocal crosstalk with disease progression, which can be helpful in promoting PCa diagnosis and therapy.