Objective: The aim of this study was to clarify the correlation between image classification and degree of pathological invasion of portal system, and evaluate the prognostic impact in pancreatic cancer. Methods: Between January 2003 and June 2017, a total of 324 resected pancreatic cancer patients (head; 244, body and tail; 80) were enrolled. As image findings, portal vein (PV) invasion was classified into type A (absent), B (unilateral narrowing), C (bilateral narrowing), and D (stenosis or obstruction with collaterals), whereas, splenic vein (SPV) invasion was classified as type α (absent), β (stenosis), and γ (obstruction). Pathological grade of portal system invasion was classified into: grade 0 (no invasion), 1 (tunica adventitia), 2 (tunica media), and 3 (tunica intima). Results: Both in PV and SPV invasions, there was a significant correlation between image classification and pathological grade (PV; P < 0.0001, SPV; P < 0.0001, respectively). In head cancer, type A had a significant better survival than type ≥ B (P < 0.0001). Whereas in body and tail cancer, there was no difference among type α, β and γ. In multivariate analysis, type B (P = 0.0003), C (P = 0.007) in head cancer and lymph node metastases (P = 0.0003) in body and tail cancer were the independent prognostic factors. Conclusion: Image classification was correlated with pathological grade in portal system invasion in pancreatic cancer. Image classification in PV invasion can be used for prognostic prediction, but not in SPV.