MRI is crucial in staging patients with rectal cancer and planning treatment. The aim was to analyse the prognostic role of MRI-predicted tumour deposits and/or extramural vascular invasion (mrTD/EMVI) in a cohort of patients with rectal cancer undergoing surgical resection, with selective neoadjuvant chemoradiotherapy (nCRT). Retrospective analysis of a single-centre cohort of consecutive patients with rectal cancer undergoing low anterior resection or abdominoperineal excision between 2008 and 2020. Unit policy was selective nCRT for MRI-predicted threatened or involved circumferential resection margin (mrCRM), or radiologically involved pelvic sidewall nodes. The primary outcome was disease-free survival. Secondary outcomes were rates of local recurrence, distant recurrence and overall survival. A total of 314 patients were analysed. Median age was 65 years (female/male: 114/200). A total of 54/314 (17%) had nCRT and 35 patients (11%) underwent abdominoperineal excision. Median follow-up was 64 months. Overall, local recurrence was detected in 18/314 (5.7%) and distant recurrence in 45/314 (14.3%). In patients not receiving nCRT (n = 260), local recurrence was detected in 11/260 (4.2%) and distant recurrence in 35/260 (13.5%). Disease-free survival was 80.5% at 5 years. Specifically, disease-free survival was 89% in mrTD/EMVI-negative and mrCRM-negative, 67% in mrTD/EMVI-positive and mrCRM-negative, and 64% in the mrCRM-positive rectal cancer (log-rank, P < 0.001). On multivariable Cox-regression analysis mrTD/EMVI was the only MRI variable associated with disease-free survival (hazard ratio 2.95; P < 0.001). mrTD/EMVI is a major prognostic indicator. Rectal cancer patients with mrCRM-negative and mrTD/EMVI-negative have excellent long-term outcomes with surgery alone. Patients with mrTD/EMVI-positive should be selectively stratified for neoadjuvant treatments in future clinical trials.