The clinical significance of peripheral blood parameters has been considered to be a potential prognostic indicator for malignancies. In this study, 224 colorectal cancer (CRC; ncolon = 103; nrectal = 121) patients who underwent resection were enrolled, and the pre- and post-operative clinical laboratory data within 1 week, before and after surgery, were collected. The prognostic value of the counts of white blood cell (WBC), neutrophil, lymphocyte and platelet, the neutrophil to lymphocyte ratio (NLR), and systemic immune-inflammation index (SII) were analyzed. Data revealed that pre-operative lymphocyte count (pre-LC) was much higher than that of post-LC (p < 0.001), and only rectal cancer patients with pre-LChigh (>median: 1.61 × 109/L) had a significantly better overall survival (OS) and 5-year survival rate (SR) than those with pre-LClow (OS: 62.3 vs. 49.5 months; SR: 74.0 vs. 43.0%; p = 0.006). Cox's proportional hazard models revealed that pre-LChigh was an independent, favorable prognostic factor for rectal cancer patients (hazard ratio = 0.348; p = 0.003). Moreover, when the disease stages were stratified, the pre-LChigh was significantly associated with better prognosis of rectal cancer patients with stage I + II rectal cancer (n = 65; OS: 67.5 vs. 54.3 months; p = 0.011). Taken together, our study revealed that pre-operative lymphocyte count is an independent prognostic factor for patients with stage I and II rectal cancer.