Abstract There is an urgent clinical need to identify patients likely to benefit from immune checkpoint inhibitor ICI treatment. Approaches available in the clinic today, such as PD-L1 immunohistochemistry (IHC) and tumor mutation burden (TMB), are insufficient for this task, in part as differences in microenvironments expressed by individual tumors may lead to heterogeneous response patterns. Recent efforts exploring the utility of quantitative imaging (radiomic) biomarkers to predict response to ICIs have shown promise to provide a more accurate and scalable method. In contrast to previously published models, our work focuses on generalizable models for predicting individual lesion-level as well as patient-level response at 3-month follow-up per RECIST criteria, using a large multi-institutional “real-world” dataset. The models combine radiomics features with demographic, molecular, and laboratory values routinely available in patients’ electronic medical records. We analyzed radiomic characteristics of 6,295 primary and metastatic lesions from 1,206 metastatic NSCLC patients treated with anti-PD-1/anti-PD-L1 ICIs from 8 institutions across the US and Europe. Patients with unavailable PD-L1 IHC, imaging follow-up, or with oncogenic driver mutations were excluded from analysis, resulting in a total dataset of 766 subjects randomly assigned to training (N=514) and validation sets (N=252). Using gradient-boosted decision tree algorithms, we developed a multi-modal predictive model to identify patients responding to ICI therapy at 3-months and evaluated its performance against an imaging-only CT radiomics model and the clinical standard of care, biopsy-based PD-L1 IHC. The multi-modal model contains CT radiomic features capturing lesion heterogeneity and spicularity, patient demographics, PD-L1 TPS, and tumor burden volume in the lung, lymph nodes, and the liver. Under the two-tailed DeLong test, the multi-modal model demonstrated statistically significant benefit over the current standard of care (PD-L1 IHC) in predicting multi-lesion 3-month response: 0.81 (P=.005) area under the receiver operating characteristic curve (ROC-AUC) in first-line ICI monotherapy patients, 0.72 (P=.044) in all-lines ICI monotherapy, and 0.71 (P=.025) in all-lines ICI-chemotherapy combination. The imaging-only model demonstrated predictive performance comparable to PD-L1 IHC: 0.71 (P=.226), 0.61 (P=.905), 0.62 (P=.674) on the same cohorts respectively. A multi-modal CT radiomics-based approach demonstrated predictive accuracy benefit over the current clinical standard and may provide an opportunity for more personalized patient management, such as risk-based escalation/de-escalation of concurrent chemotherapy in NSCLC patients. We will evaluate this methodology in prospective studies. Citation Format: Ravi B. Parikh, Petr Jordan, Rita J. Ciaravino, Ryan A. Beasley, Arpan A. Patel, Dwight H. Owen, Arya Amini, Brendan D. Curti, Ray Page, Aurelie Swalduz, Jean-Paul Beregi, Jan Chrusciel, Eric Snyder, Pritam Mukherjee, Heather M. Selby, Soohee Lee, Roshanthi Weerasinghe, Shwetha Pindikuri, Jakob B. Weiss, Andrew L. Wentland, Anish Kirpalani, An Liu, Olivier Gevaert, George Simon, Hugo JWL Aerts. Multi-institutional validation of a radiomics-based artificial intelligence method for predicting response to PD-1/PD-L1 immune checkpoint inhibitor (ICI) therapy in stage IV NSCLC. [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2023; Part 1 (Regular and Invited Abstracts); 2023 Apr 14-19; Orlando, FL. Philadelphia (PA): AACR; Cancer Res 2023;83(7_Suppl):Abstract nr 5618.