Vigorous physical activity has been associated with lower risk of fatal prostate cancer. However, mechanisms contributing to this relationship are not understood. We studied 117 men with prostate cancer in the University of North Carolina Cancer Survivorship Cohort (UNC CSC) who underwent radical prostatectomy, and 101 radiation-treated prostate cancer patients in FASTMAN. Structured questionnaires administered in UNC CSC assessed physical activity. In both studies, digital image analysis of H&E-stained tissues was applied to quantify Tumor Infiltrating Lymphocytes (TILs) in segmented regions. Nanostring gene expression profiling in UNC CSC and microarray in FASTMAN were performed on tumor tissue and a 50-gene signature utilized to predict immune cell types. Vigorous recreational activity, reported by 34 (29.1%) UNC men, was inversely associated with TILs abundance. Tumors of men reporting any vigorous activity versus none showed lower gene expression-predicted abundance of Th, exhausted CD4 T cells and macrophages. T cell subsets, including Treg, Th, Tfh, exhausted CD4 T cells, and macrophages were associated with increased risk of biochemical recurrence, only among men with ERG-positive tumors. Vigorous activity was associated with lower prostate tumor inflammation and immune microenvironment differences. Macrophages and T cell subsets, including those with immunosuppressive roles and those with lower abundance in men reporting vigorous exercise, were associated with worse outcomes in ERG-positive prostate cancer. Our novel findings contribute to our understanding of the role of the tumor immune microenvironment in prostate cancer progression, and may provide insight into how vigorous exercise could affect prostate tumor biology.