Radiotherapy (RT)-induced lymphopenia may hinder the anti-tumor immune response. Preoperative RT or chemo-RT (CRT) for locally advanced rectal cancer is a standard therapeutic approach, while immunotherapy has been approved for mismatch repair-deficient rectal tumors. We retrospectively analyzed 98 rectal adenocarcinoma patients undergoing neoadjuvant CRT with VMAT (groups A, B, C) or IMRT (group D) techniques, with four different RT schemes: group A (n = 24): 25 Gy/5 Gy/fraction plus a 0.2 Gy/fraction rectal tumor boost; group B (n = 22): 34 Gy/3.4 Gy/fraction, with a 1-week treatment break after the first five RT fractions; group C (n = 20): 46 Gy/2 Gy/fraction plus a 0.2 Gy/fraction rectal tumor boost; group D (n = 32): 45 Gy/1.8 Gy/fraction followed by 5.4 Gy/1.8 Gy/fraction to the rectal tumor. We examined the effect of the time-corrected normalized total dose (NTD-T) to the BM on lymphopenia. Groups A and B (hypofractionated RT) had significantly higher lymphocyte counts (LCs) after RT than groups C and D (p < 0.03). An inverse association between the LCs after RT and NTD-T was demonstrated (p = 0.01). An NTD-T threshold of 30 Gy delivered to 30% of the BM volume emerged as a potential constraint for RT planning, which could be successfully integrated in the RT plan. Hypofractionated and accelerated RT schemes, and BM-sparing techniques may reduce lymphocytic damage and prove critical for immuno-RT clinical trials.