Modern radiotherapy treatments can be optimized using dose-volume constraints which specify the volume of tumor and organs-at-risk receiving a given threshold dose. Careful derivation and evaluation of rectal constraints is essential to allow safe dose escalation in radiotherapy of prostate cancer. The authors present a new type of hybrid dosimetric constraint which comprises both volumetric and spatial factors of the dose-distribution. The authors also propose a framework to evaluate these constraints. The authors used data from the RT01 prostate radiotherapy trial (ISRCTN 47772397) to derive this set of hybrid constraints for the rectum based on measures extracted from dose-surface maps. For comparison, the authors also derive a set of dose-volume constraints. In order to evaluate these dosimetric constraints, the authors propose a new framework for predicting radiation-induced toxicities using Bayesian logistic regression with high-order interactions. The predictive power of the new RT01-based constraints, as well as of two sets of rectal dose-volume constraints proposed in the recent literature-The constraints proposed by other researchers [C. Fiorino, G. Fellin, T. Rancati, V. Vavassori, C. Bianchi, V. C. Borca, G. Girelli, M. Mapelli, L. Menegotti, S. Nava, and R. Valdagni, "Clinical and dosimetric predictors of late rectal syndrome after 3D-CRT for localized prostate cancer: Preliminary results of a multicenter prospective study," Int. J. Radiat. Oncol., Biol., Phys. 70, 1130-1137 (2008)] and the constraints used in the conventional or hypofractionated high dose intensity modulated radiotherapy for prostate cancer (CHHiP) trial [C. P. South, V. S. Khoo, O. Naismith, A. Norman, and D. P. Dearnaley, "A comparison of treatment planning techniques used in two randomised UK external beam radiotherapy trials for localised prostate cancer," Clin. Oncol. (R Coll. Radiol) 20, 15-21 (2008)]--were evaluated using a tenfold cross-validation with follow-up data from the RT01 trial. The predictive power was quantified using receiver-operator characteristic (ROC) curves. Toxicities considered were rectal bleeding, loose stools, and a global toxicity score. Dose-volume constraints had less predictive power than the new type of hybrid constraints. A probabilistic model for predicting rectal bleeding based on the dose-volume constraints proposed by other researchers [C. Fiorino, G. Fellin, T. Rancati, V. Vavassori, C. Bianchi, V. C. Borca, G. Girelli, M. Mapelli, L. Menegotti, S. Nava, and R. Valdagni, "Clinical and dosimetric predictors of late rectal syndrome after 3D-CRT for localized prostate cancer: Preliminary results of a multicenter prospective study," Int. J. Radiat. Oncol., Biol., Phys. 70, 1130-1137 (2008)], the CHHiP dose-volume constraints, the RT01-based dose-volume constraints, and the hybrid constraints resulted in average areas under the ROC curves of 0.56, 0.58, 0.62, and 0.67, respectively. For predicting loose stools, the corresponding values were 0.57, 0.53, 0.66, and 0.71, respectively. The areas under the respective ROC curves for predicting the global toxicity score were 0.58, 0.55, 0.61, and 0.63. Thus, imposing the new type of hybrid constraints when generating a treatment plan should result in a reduction in the incidence of radiation-induced late rectal toxicity.