Colorectal cancer (CRC) is a highly heterogeneous cancer. The molecular and cellular characteristics differ between the colon and rectal cancer type due to the differences in their anatomical location and pathological properties. With the advent of single-cell sequencing, it has become possible to analyze inter- and intra-tumoral tissue heterogeneities. A comprehensive CRC immune atlas, comprising 62,398 immune cells, was re-structured into 33 immune cell clusters at the single-cell level. Further, the immune cell lineage heterogeneity of colon, rectal, and paracancerous tissues was explored. Simultaneously, we characterized the TAM phenotypes and analyzed the transcriptomic factor regulatory network of each macrophage subset using SCENIC. In addition, monocle2 was used to elucidate the B cell developmental trajectory. The crosstalk between immune cells was explored using CellChat and the patterns of incoming and outgoing signals within the overall immune cell population were identified. Afterwards, the bulk RNA-sequencing data from The Cancer Genome Atlas (TCGA) were combined and the relative infiltration abundance of the identified subpopulations was analyzed using CIBERSORT. Moreover, cell composition patterns could be classified into five tumor microenvironment (TME) subtypes by employing a consistent non-negative matrix algorithm. Finally, the co-expression and interaction between SPP1+TAMs and Treg cells in the tumor microenvironment were analyzed by multiplex immunohistochemistry. In the T cell lineage, we found that CXCL13+T cells were more widely distributed in colorectal cancer tissues, and the proportion of infiltration was increased. In addition, Th17 was found accounted for the highest proportion in CD39+CD101+PD1+T cells. Mover, Ma1-SPP1 showed the characteristics of M2 phenotypes and displayed an increased proportion in tumor tissues, which may promote angiogenesis. Plasma cells (PCs) displayed a significantly heterogeneous distribution in tumor as well as normal tissues. Specifically, the IgA+ PC population could be shown to be decreased in colorectal tumor tissues whereas the IgG+ PC one was enriched. In addition, information flow mediated by SPP1 and CD44, regulate signaling pathways of tumor progression. Among the five TME subtypes, the TME-1 subtype displayed a markedly reduced proportion of T-cell infiltration with the highest proportion of macrophages which was correlated to the worst prognosis. Finally, the co-expression and interaction between SPP1+TAMs and Treg cells were observed in the CD44 enriched region. The heterogeneity distribution and phenotype of immune cells were analyzed in colon cancer and rectal cancer at the single-cell level. Further, the prognostic role of major tumor-infiltrating lymphocytes and TME subtypes in CRC was evaluated by integrating bulk RNA. These findings provide novel insight into the immunotherapy of CRC.