Abstract Background Tertiary lymphoid structures (TLS) are major component in immune microenvironment, correlating with a favorable prognosis in colorectal cancer. However, the methods used to define and characterize TLS were not united, hindering its's clinical application. This study aims to seek a more stable method to characterize TLS and clarify its prognostic value in larger multicenter cohorts. Methods A total of 1609 patients from four hospitals and the TCGA database were analyzed. We quantified the number and maximum length of TLS along the invasive margin of tumor using hematoxylin and eosin (HE)-stained whole-slide images (WSIs). Additionally, the length of the invasive margin was determined to calculate the TLS density. The prognostic value of the TLS for overall survival was evaluated. In addition, we examined the association between TLS density and immune cell infiltration using immunohistochemistry-stained WSIs. The performance for predicting overall survival was measured using hazard ratios (HR) with 95% confidence intervals (CI). Results Among the three TLS quantification methods, TLS density has the strongest discriminative performance. Survival analysis indicated that higher TLS density correlated with better overall survival (HR for high vs. low 0.57 [95% CI 0.42–0.78] in primary cohort; 0.49 [0.35–0.69] in validation cohort; 0.35 [0.18–0.67] in TCGA cohort). A high TLS density was associated with a high level of CD3+ T cells infiltration. Conclusions Based on this comparative multicenter analysis, TLS density was identified as a simple, robust, and effective immune prognostic index for colorectal cancer.