Increasing evidence highlights the clinical significance of stromal cells and immune cells in the liver cancer microenvironment. However, reliable prognostic models have not been well established. This study aimed to develop a gene signature for liver cancer based on stromal and immune scores. Using the estimation of stromal and immune cells in malignant tumor tissues using expression data (ESTIMATE) algorithm, stromal and immune scores were estimated based on the transcriptome profile of The Cancer Genome Atlas (TCGA) liver cancer cohort. Stromal-/immune-related differentially expressed genes were identified, followed by functional enrichment analysis. The Cox regression model was used to select prognostic genes and construct a gene signature. Its predictive potential was evaluated by receiver operating characteristic (ROC). The correlation between the risk score and immune cell infiltration was analyzed using Tumor Immune Estimation Resource (TIMER). Three hundred sixty-four upregulated and 10 downregulated stromal-/immune-related genes were identified, were mainly enriched in immune-related processes and pathways. Through univariate and multivariate cox survival analysis, a five-gene risk score was constructed, composed of FABP3, HTRA3, OLFML2B, PDZD4 and SLAMF6. Patients with high score indicated a poorer prognosis than those with low risk score. The areas under the ROC curves of overall survival (OS), progression-free interval, 3-, 5-year, OS status were 0.68, 0.57, 0.72, 0.74 and 0.728, indicating its well performance on predicting patients' prognoses. Furthermore, the risk score and the five genes were significantly correlated with immune cell infiltration in the tumor microenvironment. In this study, we proposed a prognostic five-gene signature based on stromal/immune scores in the liver cancer microenvironment.