The aim of this study is to clarify the association between lymphovascular invasion (LVI) and/or perineural invasion (PNI) and the clinical characteristics and prognostic importance of rectal cancer, to provide a basis for early adjuvant treatment of rectal cancer. We retrospectively analyzed patients diagnosed with rectal cancer. This study involved rectal cancer tissue samples were obtained by surgical methods. Data on histological form, tumor classification, tumor size, gross growth pattern, blood and lymphatic vessel invasion, and PNI of the slice by HE staining were obtained from pathological examination. Immunohistochemical analysis of tissue samples was performed to determine p53 and EGFR expressions. There were 330 rectal cancer patients included in the study. LVI and/or PNI can be used as a high-risk factor for the prognosis of rectal cancer, predict prognostic survival, and guide adjuvant therapy. The detection rates of LVI and PNI were 32.1% and 16.1%. Differentiation grade, Union for International Cancer Control staging, tumor-lymph node-metastasis staging are significantly related to LVI or PNI. Multivariate logistic regression analysis shows that poor differentiation and N ≥ 1 can be used as independent risk factors and predictive factors for LVI. At the same time, poor differentiation and T > 3 is an independent risk factor for PNI. Only poor differentiation is the risk factor for poor prognosis in Cox risk regression analysis. In addition, the simultaneous occurrence of LVI and PNI is an independent prognostic factor.