Predicting clinical outcomes in the S1314-COXEN trial using a multimodal deep learning model integrating histopathology, cell types, and gene expression.

Abstract

533 Background: Accurate prediction of response to neoadjuvant chemotherapy (NAC) is essential for optimizing treatment outcomes in muscle-invasive bladder cancer (MIBC). We present a cutting-edge multimodal deep learning model aimed at predicting outcomes of NAC from accessible H & E images and molecular data. Methods: We designed a Graph-based Multi-modal Late Fusion (GMLF) deep learning model that integrates three types of data from the S1314-COXEN clinical trial: 1) Neural embeddings from 182 whole slide images (WSIs), generated via the ResNet50 architecture. 2) Cell types based on morphology, including cancer, necrotic, immune, and stromal cells, and their spatial positions within the tumor microenvironment extracted from WSIs using the HoVer-Net framework, a multiple-branch convolutional neural network for predicting cell types. 3) Patient-level RNA expression data derived from 1,071-dimensional gene expression vectors. The dataset was randomly divided into 80% for model training, using 5-fold cross-validation (5-fold CV), and 20% for internal validation. Results: Our dataset included 182 WSIs of 180 patients who were randomized to receive either gemcitabine-cisplatin or dose-dense methotrexate-vinblastine-adriamycin-doxorubicin-cisplatin. Of all patients, 30.8% demonstrated a complete pathological response (pT0 after radical cystectomy). Our GMLF model achieved an AUROC of 0.7417 ± 0.1021 for predicting whether a patient showed a complete pathologic response to NAC in a 5-fold CV. Using an 80/20 training and validation split, the model yielded an AUROC of 0.7236. Interestingly, our model emphasized the contribution of spatial information collected from WSIs. For selecting the best architecture for the histology branch, the Slidegraph+ framework, a graph neural network-based model, achieved an AUROC of 0.6938 ± 0.0565, significantly outperforming patch-level models as the second-best, which achieved AUROC of 0.5572 ± 0.1793. Shapley Additive Explanation (SHAP) analysis revealed RNA expression branch as the most influential, with a mean SHAP magnitude of 0.13, followed by the neural embeddings at 0.12. SHAP-based interpretation identified the most important expressed genes impacting model predictability, including TP63. Our gene set analysis identified the most influential pathways for predicting response, which included basal differentiation and myo-fibroblast enrichment (p-value < 0.05). Conclusions: Our interpretable GMLF model accurately predicts NAC response in patients with MIBC by integrating standard H&E images and RNA expression data. Our model interpretations not only revealed the importance of each modality but also uncovered histopathological, cellular, and molecular underpinnings of response to NAC in MIBC. These findings open the door to AI-guided development of precision therapies for MIBC. Clinical trial information: NCT02177695 .