This study compared survival and metastasis occurrence between colorectal neuroendocrine neoplasms (cNEN) and colorectal adenocarcinoma with neuroendocrine differentiation (cNED) and further explored their prognostic factors and treatment indicators. Patients diagnosed as cNEN and cNED in West China Hospital from January 2009 to December 2020 were enrolled. The diagnosis and metastasis rates were calculated. Univariate and multivariate Cox analyses were conducted for progression-free survival (PFS) in cNEN surgical patients, and generalized linear regression was used for metastatic disease. The study enrolled 435 patients, including 257 neuroendocrine tumors (NET), 52 neuroendocrine carcinomas (NEC), 29 mixed neuroendocrine-non-neuroendocrine neoplasms (MiNEN), and 97 NED patients, of whom 202 received local resection, and 233 received radical resection. Metastasis rates were higher in MiNEN and NEC groups compared to other groups (NED: 28.9%, MiNEN: 58.6%, NEC: 65.4%, NET: 8.6%, p < 0.001). The liver is the main metastatic site in cNEN, whereas cNED metastasized to various sites. For NEC and MiNEN patients, colon location (p = 0.002) and T stage > 2 (p = 0.040) were associated with disease progression separately. Independent risk factors for metastatic NET included tumor grade G2/G3 (p < 0.001), colon location (p = 0.001), size ≥ 1cm (p = 0.005), and CK20 partial positive (p < 0.001). cNEN show high metastatic capacity and are challenging to diagnose. More aggressive treatment and follow-up strategies are necessary for those patients. NET tumor grade higher than G2, size larger than 1cm, or located in the colon should be managed with radical surgery.