Tumor depth evaluation is essential for pathological tumor staging because it affects clinical management as an independent risk factor for lymph node metastasis in colorectal cancers. However, poor interobserver variability of invasion depth has been reported. This study aimed to clarify the effectiveness of desmin immunostaining in the histological diagnosis of colorectal cancer. Overall, 63 sets of slides of colorectal cancer stained with hematoxylin and eosin (H&E) and desmin were prepared and independently reviewed by four examiners. After reviewing the desmin-stained slides, the interobserver variability of H&E slides alone was significantly improved for all examiners. For the assessment of Tis vs. T1, the sensitivity and accuracy were significantly improved for all examiners by combining H&E and desmin immunostaining. For the diagnosis of T1b vs. Tis or T1a, specificity and accuracy were significantly improved by adding desmin immunostaining. Ancillary desmin staining to assess submucosal invasion in colorectal cancers significantly improved interobserver agreement, led to efficient screening of T1 cancers, and reduced excessive T1b diagnoses. The combination of desmin immunostaining and H&E staining is highly recommended for diagnosing invasive colorectal cancer.