The Immunoscore can categorize patients into high- and low-risk groups for prognostication in colorectal cancer (CRC). Collagen plays an important role in immunomodulatory functions in the tumor microenvironment (TME). However, the correlation between collagen and the Immunoscore in the TME is unclear. This study aimed to construct a collagen signature to illuminate the relationship between collagen structure and Immunoscore. A total of 327 consecutive patients with stage I-III stage CRC were included in a training cohort. The fully quantitative collagen features were extracted at the tumor center and invasive margin of the specimens using multiphoton imaging. LASSO regression was applied to construct the collagen signature. The association of the collagen signature with Immunoscore was assessed. A collagen nomogram was developed by incorporating the collagen signature and clinicopathological predictors after multivariable logistic regression. The performance of the collagen nomogram was evaluated via calibration, discrimination, and clinical usefulness and then tested in an independent validation cohort. The prognostic values of the collagen nomogram were assessed using Cox regression and the Kaplan-Meier method. The collagen signature was constructed based on 16 collagen features, which included 6 collagen features from the tumor center and 10 collagen features from the invasive margin. Patients with a high collagen signature were more likely to show a low Immunoscore (Lo IS) in both cohorts (P<0.001). A collagen nomogram integrating the collagen signature and clinicopathological predictors was developed. The collagen nomogram yielded satisfactory discrimination and calibration, with an AUC of 0.925 (95% CI: 0.895-0.956) in the training cohort and 0.911 (95% CI: 0.872-0.949) in the validation cohort. Decision curve analysis confirmed that the collagen nomogram was clinically useful. Furthermore, the collagen nomogram-predicted subgroup was significantly associated with prognosis. Moreover, patients with a low-probability Lo IS, rather than a high-probability Lo IS, could benefit from chemotherapy in high-risk stage II and stage III CRC patients. The collagen signature is significantly associated with the Immunoscore in the TME, and the collagen nomogram has the potential to individualize the prediction of the Immunoscore and identify CRC patients who could benefit from adjuvant chemotherapy.