BackgroundDeep learning techniques excel at identifying tumor-infiltrating lymphocytes (TILs) and immune phenotypes in hematoxylin and eosin (H&E)-stained slides. However, their ability to elucidate detailed functional characteristics of diverse cellular phenotypes within tumor immune microenvironment (TME) is limited. We aimed to enhance our understanding of cellular composition and functional characteristics across TME regions and improve patient stratification by integrating H&E with adjacent immunohistochemistry (IHC) images.MethodsA retrospective study was conducted on patients with Human Papillomavirus-positive oropharyngeal squamous cell carcinoma (OPSCC). Using paired H&E and IHC slides for 11 proteins, a deep learning pipeline was used to quantify tumor, stroma, and TILs in the TME. Patients were classified into immune inflamed (IN), immune excluded (IE), or immune desert (ID) phenotypes. By registering the IHC and H&E slides, we integrated IHC data to capture protein expression in the corresponding tumor regions. We further stratified patients into specific immune subtypes, such as IN, with increased or reduced CD8+ cells, based on the abundance of these proteins. This characterization provided functional insight into the H&E-based subtypes.ResultsAnalysis of 88 primary tumors and 70 involved lymph node tissue images reveals an improved prognosis in patients classified as IN in primary tumors with high CD8 and low CD163 expression (p = 0.007). Multivariate Cox regression analysis confirms a significantly better prognosis for these subtypes.ConclusionsIntegrating H&E and IHC data enhances the functional characterization of immune phenotypes of the TME with biological interpretability, and improves patient stratification in HPV( + ) OPSCC.