Background Nucleotide excision repair (NER) is pivotal in the development of smoking-related malignancies. Nine core genes (XPA, XPB, XPC, XPD, XPF, XPG, ERCC1, DDB1, and DDB2) are highly involved in the NER process. We combined two phenotypes of NER pathway (NER protein and NER gene mRNA expression) and evaluated their associations with the risks of the head and neck squamous cell carcinomas (HNSCCs) in a Chinese population. Methods We conducted a case-control study of 337 HNSCC patients and 285 cancer-free controls by measuring the expression levels of nine core NER proteins and NER gene mRNA in cultured peripheral lymphocytes. Results Compared with the controls, cases had statistically significantly lower protein expression levels of XPA (P < 0.001) and lower mRNA expression levels of XPA and XPB (P = 0.005 and 0.001, respectively). After dividing the subjects by controls' medians of expression levels, we found an association between increased risks of HNSCCs and low XPA protein level (Ptrend = 0.031), as well as low mRNA levels of XPA and XPB (Ptrend = 0.024 and 0.001, respectively). Subsequently, we correlated the two phenotypes and found associations between the NER mRNA and protein levels. Finally, the sensitivity of the expanded model with protein and mRNA expression levels, in addition to demographic variables, on HNSCCs risk was significantly improved. Conclusions Combining two phenotypes of NER pathway may be more effective than the model only including one single phenotype for the assessment of risks of HNSCCs.