Tumor mutation burden (TMB) has been associated with prognosis in various malignancies, but it has yet to be elucidated in hepatocellular carcinoma (HCC). We aimed to investigate the prognostic effects of TMB and its relationship with immune infiltration through multiple databases and whole-exome sequencing, so as to establish a panel model capable of predicting prognosis. The results demonstrated that the prognosis of high TMB group was worse than that of low TMB group, with a cutoff TMB value of 4.9. Enrichment analysis demonstrated that differentially expressed genes were mainly related to T cell activation, cell membrane localization and matrix composition. Tumor immune infiltration analysis revealed the infiltrations of Th2, Th17, and Tgd were up-regulated in high TMB group, while those of Tr1, MAIT, and DC were up-regulated in low TMB group. TMB-Infiltration model fit well with the actual survival observation, with a C-index 0.785 (0.700-0.870), which verified in ICGC-LIRI-JP was 0.650 (0.553-0.747). Additionally, these screened immune genes performed well in predicting tumor vascular invasion with a C-index of 0.847 (0.778-0.916). Overall, these results indicated that patients with high mutation frequency of immune-related genes and high TMB were prone to have worse prognosis and relapse after radical treatment.