Systematic classification of colon and rectal cancer-associated signatures is critical for the classification and prognosis of cancer patients. In this study, we identified a panel of 29 colon and rectal cancer-associated signatures from bioinformatics analyses on both TCGA and GEO datasets. Based on the signatures, we developed a machine learning method to classify colon and rectal cancer into three immune subtypes named High-Immunity Subtype, Medium-Immunity Subtype, and Low-Immunity Subtype respectively. Reconfirmed by different datasets, this classification was associated with the tumor mutational burden (TMB) and many cancer-associated pathways. Compared to Medium-Immunity and Low-Immunity, patients with High-Immunity Subtype have a greater immune cell infiltration and better survival prognosis. In addition, a prognostic signature of six differentially‐expressed and survival-associated genes among the three cancer subtypes (CERCAM, CD37, CALB2, MEOX2, RASGRP2 and PCOLCE2) was identified by the multivariable COX analysis, which was further used to develop an accurate model to predict the prognosis of colon and rectal cancer patients.