Introduction SBRT is an increasingly popular treatment for localized prostate cancer, though considerable variation in technical approach is common and optimal dose constraints are uncertain. In this study, we sought to identify dosimetric and patient-related predictors of acute rectal toxicity. Methods Patients included in this study were treated with prostate SBRT on a prospective institutional protocol. Physician-graded toxicity and patient-reported outcomes were captured at one week, one month, and three months following SBRT. DVH data were extracted and converted into relative volume differential DVHs for NTCP modeling. Patient- and disease-related covariates along with NTCP model predictions were independently tested for significant association with physician-graded toxicity or a decline in bowel-related QoL. A multivariate model was constructed using forward selection, and significant parameter cutoff values were obtained with Fischer’s exact test to group patients by risk of developing physician-graded toxicity or detriments in patient-reported QoL. Results One hundred and three patients treated for localized prostate cancer with SBRT were included in our analysis. 52% of patients experienced a clinically significant decline in bowel-related QOL within 1 week of completion of treatment, while only 27.5% of patients developed grade 2+ physician-graded rectal toxicity. Sequential feature selection multivariate logistic regression identified rectal V22.5 Gy (p = 0.001) and D19% (p = 0.001) as independent predictors of clinically significant toxicity, while rectal V20Gy (p = 0.004) and D25.3% (p = 0.007) were independently correlated with physician-graded toxicity. Global multivariate step-wise logistic regression identified only D19% (p = 0.001) and V20Gy (p = 0.004) as independent predictors of acute bowel bother or physician-graded rectal toxicity respectively. Conclusions Moderate doses to large rectal volumes, D19% and V20Gy, were associated with an increased incidence of a clinically significant decrease in patient-reported bowel QOL and physician-scored grade 2+ rectal toxicity, respectively. These dosimetric parameters may help practitioners mitigate acute toxicity in patients treated with prostate SBRT.