Background High endothelial venules (HEV) and tertiary lymphoid structures (TLS) are associated with clinical outcomes of patients with colorectal cancer (CRC). However, because HEV are components of TLS, there have been few studies of the role of the HEV proportion in TLS (HEV/TLS). This study investigated the role of the HEV/TLS and its relationship with the tumor immune microenvironment in CRC. Methods A retrospective analysis of 203 cases of tissue pathologically diagnosed as CRC after general surgery was performed at the First Affiliated Hospital of Jinan University from January 2014 to July 2017. Paraffin sections were obtained from the paracancerous intestinal mucosal tissues. The area of HEV and TLS and immune cells were detected by immunohistochemistry. We further divided the positive HEV expression group into the high HEV/TLS group and the low HEV/TLS group by the average area of HEV/TLS. After grouping, the data were also analyzed using the chi-square test, Kaplan-Meier method, and univariate and multivariate Cox proportional risk regression analyses. A correlation analysis of the HEV/TLS and immune cells as well as angiogenesis was performed. Results Patients with a high HEV/TLS in CRC tissue were associated with longer OS, DFS and lower TNM stage. Meanwhile, CRC tissue with a high HEV/TLS showed a greater ability to recruit the CD3+ T cells, CD8+ T cells and M1 macrophages and correlated with less angiogenesis. Conclusively, high HEV/TLS links to the favorable prognosis of CRC patients and correlated with anti-tumor immune microenvironment, which can be a potential biomarker for prognosis of CRC patients. Conclusion A high HEV/TLS is associated with a favorable prognosis for CRC and is correlated with the anti-tumor immune microenvironment. Therefore, it is a potential biomarker of the CRC prognosis. KEY MESSAGES High HEV/TLS is associated with a favorable prognosis for CRC. High HEV/TLS correlated with the anti-tumor immune microenvironment of CRC and can serve as a novel prognostic biomarker.