To explore the value of multi-parametric MRI (mp-MRI) radiomic model for preoperative prediction of recurrence and/or metastasis (RM) as well as survival benefits in patients with rectal cancer. A retrospective analysis of 234 patients from two centers with histologically confirmed rectal adenocarcinoma was conducted. All patients were divided into three groups: training, internal validation (in-vad) and external validation (ex-vad) sets. In the training set, radiomic features were extracted from T2WI, DWI, and contrast enhancement T1WI (CE-T1) sequence. Radiomic signature (RS) score was then calculated for feature screening to construct a rad-score model. Subsequently, preoperative clinical features with statistical significance were selected to construct a clinical model. Independent predictors from clinical and RS related to RM were selected to build the combined model and nomogram. After feature extraction, 26 features were selected to construct the rad-score model. RS (OR = 0.007, p < 0.01), MR-detected T stage (mrT) (OR = 2.92, p = 0.03) and MR-detected circumferential resection margin (mrCRM) (OR = 4.70, p = 0.01) were identified as independent predictors of RM. Then, clinical model and combined model were constructed. ROC curve showed that the AUC, accuracy, sensitivity and specificity of the combined model were higher than that of the other two models in three sets. Kaplan-Meier curves showed that poorer disease-free survival (DFS) time was observed for patients in pT3-4 stages with low RS score (p < 0.001), similar results were also found in pCRM-positive patients (p < 0.05). The mp-MRI radiomics model can be served as a noninvasive and accurate predictors of RM in rectal cancer that may support clinical decision-making.