Performance Of Prognostic Models Research Articles

Discrimination and calibration are insensitive metrics for comparing the performance of cardiovascular risk prediction models

Abstract Background/Introduction Accurate cardiovascular disease (CVD) risk prediction models can facilitate the identification and cost-effective treatment of high-risk individuals. Discrimination and calibration are the most commonly recommended metrics for evaluating model performance. However, both are population-level measures, and it remains unclear whether these metrics are sufficiently sensitive to assess the performance of models designed for use with individual patients. Purpose Using a real-world dataset, we aimed to compare CVD risk estimates for example individuals (individual-level measures) and discrimination and calibration metrics (population-level measures) from two CVD risk prediction models; one with "age" as the only predictor and one with multiple predictors. Methods We constructed a cohort of almost all New Zealanders without CVD or heart failure, alive and aged 30–79 years in 2014, with follow-up linkage to hospitalisations and mortality until 2018 (N = 2,098,359). We derived two sets of sex-specific Cox regression models to predict 5-year CVD risk. The age-only models included "age" as the only predictor, and the full models used nine pre-defined routinely available CVD risk predictors with interaction terms. We calculated and compared individuals’ 5-year CVD risk from the age-only and full models and assessed model performance using Harrell’s C-statistics and calibration plots. Results Risk estimates for individuals using the two models showed substantial differences. A 65-year-old woman had predicted 5-year risks of 4.9% with the age-only model but 16.1% with the full model; a 74-year-old man had predicted risks of 15.2% and 9.2%, respectively, with age-only and full models (Figure 1). However, the models had similar discrimination and calibration (Figure 2). Age-only models had good discrimination with C-statistics of 0.773 (95% CI: 0.771–0.775) in women and 0.740 (0.738–0.742) in men, and calibrated well in both sexes. The full models had slightly better discrimination (C-statistics: 0.814 [0.812–0.816] in women and 0.769 [0.767–0.771] in men), and very similar calibration to the age-only models in both sexes. Conclusions Age-only models showed good discrimination and were well calibrated, but predicted very different risks in individual case studies compared to models with multiple predictors. This study calls into question the use of discrimination and calibration metrics as the main measures for comparing the performance of different CVD risk prediction models. We recommend that examples of predicted risk for individuals and an assessment of the magnitude and prevalence of cardiovascular predictors should also be considered when comparing the performance of prognostic models.

The performance of prognostic models depended on the choice of missing value imputation algorithm: a simulation study

ObjectiveThe development of clinical prediction models is often impeded by the occurrence of missing values in the predictors. Various methods for imputing missing values before modelling have been proposed. Some of them are based on variants of multiple imputation by chained equations, while others are based on single imputation. These methods may include elements of flexible modelling or machine learning algorithms, and for some of them user-friendly software packages are available. The aim of this study was to investigate by simulation if some of these methods consistently outperform others in performance measures of clinical prediction models. Study Design and SettingWe simulated development and validation cohorts by mimicking observed distributions of predictors and outcome variable of a real data set. In the development cohorts, missing predictor values were created in 36 scenarios defined by the missingness mechanism and proportion of non-complete cases. We applied three imputation algorithms that were available in R software: mice, aregImpute and missForest. These algorithms differed in their use of linear or flexible models, or random forests, the way of sampling from the predictive posterior distribution, and the generation of a single or multiple imputed data sets. For multiple imputation we also investigated the impact of the number of imputations. Logistic regression models were fitted with the simulated development cohorts before (full data analysis) and after missing value generation (complete case analysis), and with the imputed data. Prognostic model performance was measured by the scaled Brier score, c-statistic, calibration intercept and slope, and by the mean absolute prediction error evaluated in validation cohorts without missing values. Performance of full data analysis was considered as ideal. ResultsNone of the imputation methods achieved the model’s predictive accuracy that would be obtained in case of no missingness. In general, complete case analysis yielded the worst performance, and deviation from ideal performance increased with increasing percentage of missingness and decreasing sample size. Across all scenarios and performance measures, aregImpute and mice, both with 100 imputations, resulted in highest predictive accuracy. Surprisingly aregImpute outperformed full data analysis in achieving calibration slopes very close to 1 across all scenarios and outcome models. The increase of mice’s performance with 100 compared to 5 imputations was only marginal. The differences between the imputation methods decreased with increasing sample sizes and decreasing proportion of non-complete cases. ConclusionIn our simulation study, model calibration was more affected by the choice of the imputation method than model discrimination. While differences in model performance after using imputation methods were generally small, multiple imputation methods as mice and aregImpute that can handle linear or nonlinear associations between predictors and outcome are an attractive and reliable choice in most situations.

Identification and validation of genes associated with prognosis of cisplatin-resistant ovarian cancer

PurposeTo investigate the role of prognostic genes related to cisplatin resistance in ovarian cancer during disease progression.MethodThe gene expression profile of the NCI-60 cell line was acquired through comprehensive analysis of the GEO database accession GSE116439. We performed a thorough analysis of gene expression differences in samples from seven individuals exposed to cisplatin concentrations of 0 nM compared to seven samples exposed to 15000 nM over a 24-h period. Key genes were initially identified through LASSO regression, followed by their enrichment through differential gene function analysis (GO) and pathway enrichment analysis (KEGG). Subsequently, a prognostic risk model was established for these key genes. The prognostic model's performance was assessed through K-M survival curves and ROC curves. To examine the variance in immune cell infiltration between the high and low-risk groups, CIBERSORTx analysis was employed. Finally, validation of prognostic gene expression in cisplatin-resistant ovarian cancer was carried out using clinical samples, employing RT-qPCR and Western Blot techniques.ResultsA total of 132 differential genes were found between cisplatin resistance and control group, and 8 key prognostic genes were selected by analysis, namely VPS13B, PLGRKT, CDKAL1, TBC1D22A, TAP1, PPP3CA, CUX1 and PPP1R15A. The efficacy of the risk assessment model derived from prognostic biomarkers, as indicated by favorable performance on both Kaplan–Meier survival curves and ROC curves. Significant variations in the abundance of Macrophages M1, T cells CD4 memory resting, T cells follicular helper, and T cells gamma delta were observed between the high and low-risk groups. To further validate our findings, RT-qPCR and Western Blot analyses were employed, confirming differential expression of the identified eight key genes between the two groups.ConclusionVPS13B, TBC1D22A, PPP3CA, CUX1 and PPP1R15A were identified as poor prognostic genes of cisplatin resistance in ovarian cancer, while PLGRKT, CDKAL1 and TAP1 were identified as good prognostic genes. This offers a novel perspective for future advancements in ovarian cancer treatment, suggesting potential avenues for the development of new therapeutic targets.

Exploring the prognostic value and biological pathways of transcriptomics and radiomics patterns in glioblastoma multiforme

ObjectivesTo develop a multi-omics prognostic model integrating transcriptomics and radiomics for predicting overall survival in patients with glioblastoma multiforme (GBM), and investigate the biological pathways of radiomics patterns. Materials and methodsTranscription profiles of GBM patients and normal controls were used to obtain differentially expressed mRNAs and long non-coding RNAs (lncRNAs). Radiomics features were extracted from magnetic resonance imaging (MRI). Least absolute shrinkage and selection operator (LASSO) Cox regression was employed to select survival-associated features for the construction of transcriptomics and radiomics signatures. Genes associated with GBM prognosis were identified through the analysis of lncRNA-mRNA co-expression networks and Weighted Gene Co-expression Network Analysis (WGCNA), and their biological pathways were investigated using Genomes enrichment analysis. Transcriptomics, radiomics, and clinical data were integrated to evaluate the multi-omics prognostic model's performance. ResultsLASSO Cox regression yielded 21 survival-related features, including 19 transcriptomics features and 2 radiomics features. Based on transcriptomics and radiomics signature, GBM patients were classified as high-risk or low-risk. The genes obtained from the co-expression network screen were associated with microtubule binding, while those from the WGCNA screen were associated with growth factor receptor binding. In the training set, the AUC values for the multi-omics model and clinical model were 0.964 and 0.830, respectively, while in the validation set, they were 0.907 and 0.787. The multi-omics prognostic model outperformed the clinical prognostic model. ConclusionsThe co-expression network and WGCNA methods revealed genes associated with multiple biological pathways in GBM. The multi-omics prognostic model demonstrated excellent performance and indicated significant potential for clinical application.

African American race does not confer an increased risk of clinical events in patients with primary sclerosing cholangitis.

The natural history of primary sclerosing cholangitis (PSC) among African Americans (AA) is not well understood. Transplant-free survival and hepatic decompensation-free survival were assessed using a retrospective research registry from 16 centers throughout North America. Patients with PSC alive without liver transplantation after 2008 were included. Diagnostic delay was defined from the first abnormal liver test to the first abnormal cholangiogram/liver biopsy. Socioeconomic status was imputed by the Zip code. Among 850 patients, 661 (77.8%) were non-Hispanic Whites (NHWs), and 85 (10.0%) were AA. There were no significant differences by race in age at diagnosis, sex, or PSC type. Inflammatory bowel disease was more common in NHWs (75.8% vs. 51.8% p=0.0001). The baseline (median, IQR) Amsterdam-Oxford Model score was lower in NHWs (14.3, 13.4-15.2 vs. 15.1, 14.1-15.7, p=0.002), but Mayo risk score (0.03, -0.8 to 1.1 vs. 0.02, -0.7 to 1.0, p=0.83), Model for End-stage Liver Disease (5.9, 2.8-10.7 vs. 6.4, 2.6-10.4, p=0.95), and cirrhosis (27.4% vs. 27.1%, p=0.95) did not differ. Race was not associated with hepatic decompensation, and after adjusting for clinical variables, neither race nor socioeconomic status was associated with transplant-free survival. Variables independently associated with death/liver transplant (HR, 95% CI) included age at diagnosis (1.04, 1.02-1.06, p<0.0001), total bilirubin (1.06, 1.04-1.08, p<0.0001), and albumin (0.44, 0.33-0.61, p<0.0001). AA race did not affect the performance of prognostic models. AA patients with PSC have a lower rate of inflammatory bowel disease but similar progression to hepatic decompensation and liver transplant/death compared to NHWs.

Modest performance of text mining to extract health outcomes may be almost sufficient for high-quality prognostic model development

Background:Across medicine, prognostic models are used to estimate patient risk of certain future health outcomes (e.g., cardiovascular or mortality risk). To develop (or train) prognostic models, historic patient-level training data is needed containing both the predictive factors (i.e., features) and the relevant health outcomes (i.e., labels). Sometimes, when the health outcomes are not recorded in structured data, these are first extracted from textual notes using text mining techniques. Because there exist many studies utilizing text mining to obtain outcome data for prognostic model development, our aim is to study the impact of the text mining quality on downstream prognostic model performance. Methods:We conducted a simulation study charting the relationship between text mining quality and prognostic model performance using an illustrative case study about in-hospital mortality prediction in intensive care unit patients. We repeatedly developed and evaluated a prognostic model for in-hospital mortality, using outcome data extracted by multiple text mining models of varying quality. Results:Interestingly, we found in our case study that a relatively low-quality text mining model (F1 score ≈ 0.50) could already be used to train a prognostic model with quite good discrimination (area under the receiver operating characteristic curve of around 0.80). The calibration of the risks estimated by the prognostic model seemed unreliable across the majority of settings, even when text mining models were of relatively high quality (F1 ≈ 0.80). Discussion:Developing prognostic models on text-extracted outcomes using imperfect text mining models seems promising. However, it is likely that prognostic models developed using this approach may not produce well-calibrated risk estimates, and require recalibration in (possibly a smaller amount of) manually extracted outcome data.

Clinical prognostic models in children with sepsis in low- and middle-income countries: a systematic review and meta-analysis.

Sepsis is the leading cause of child death worldwide, with the majority of these deaths occurring in low- and middle-income countries (LMICs). The aim of this systematic review and meta-analysis was to describe clinical prognostic scores and models for pediatric sepsis outcomes and assess the performance of these scores for predicting mortality in LMICs. Ovid Medline, CINAHL, Cochrane Library, EBSCO Global Health, and Web of Science, were searched through September 2022 for citations related to the development or validation of a clinical prognostic score or model among children with sepsis, conducted in LMIC. Titles, abstracts, and full texts were screened by two independent reviewers and data extracted included population characteristics, variables included, outcomes, and model performance. Risk of bias was assessed with the Prediction Model Risk of Bias Assessment Tool (PROBAST). 4,251 titles/abstracts and 315 full-text studies were screened, with 12 studies meeting inclusion criteria. Study countries included India, China, Egypt, Indonesia, Tanzania, and a multi-site study in Latin America. Prognostic scores/models included existing scores such as PELOD-2, pSOFA, PRISM, P-MODS, refractory shock criteria. There was high risk of bias in all studies. Meta-analysis was possible for pSOFA, PELOD-2, PRISM, and P-MODS, with pooled area under the receiver-operator characteristic curve of 0.86 (95%CI 0.78-0.94), 0.83 (95% CI 0.76-0.91), respectively. Relatively few clinical scores and models have been externally validated for prognostication and risk-stratification among children with sepsis in diverse LMIC settings. Notably there were no studies from low-income countries. Some potentially relevant studies were excluded due to lack of clarity regarding the presence of sepsis in the study populations. More widespread and standardized use of sepsis criteria may aid in better understanding the burden of sepsis and prognostic model performance at the bedside among children in LMICs. Further research to externally validate, implement and adapt these models is needed to account for challenges in use of these scores in resource-limited settings. https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42022340126, PROSPERO [CRD42022340126].

Risk Factors and Prognostic Model for II-IV aGVHD in Patients with Engraftment Syndrome after Haploidentical Hematopoietic Stem Cell Transplantation

Background Engraftment syndrome (ES), characterized by noninfective fever, rash, noncardiogenic pulmonary edema, hepatic dysfunction and weight gain, is a common complication after haplo-HSCT. Although ES has been reported in the previous literature, there is still no consensus on the risk factors, outcomes and treatment of ES, as there is not only a lack of uniformity described by authors but also conflicting evidence between studies. One meta-analysis found that patients with ES have significantly higher odds of developing aGVHD than patients without ES. However, a strategy for identifying high-risk patients with ES for aGVHD remains unclear although important for early aGVHD prevention and the improvement of patient outcomes. Here, we retrospectively analyzed a large cohort of posthaplo-HSCT ES patients and described the risk factors, outcomes and prognostic model for II-IV aGVHD in patients with ES. Methods A total of 384 patients with ES after haplo-HSCT at Peking University People's Hospital between 2013 and 2022 were retrospectively identified. We first performed a nested case‒control study to identify the risk factors for ES, and three controls were randomly selected for each case according to the time of transplant and the length of follow-up. The cohort of ES patients was further divided into a derivation cohort of 258 patients and a validation cohort of 126 patients. We used a logistic regression model to perform univariate and multivariate analyses to determine the risk factors and prognostic factors for II-IV aGVHD in patients with ES after haplo-HSCT. The prognostic model performance was assessed by evaluating the discrimination [area under the curve (AUC)], calibration (calibration plot), and net benefit [decision curve analysis (DCA)]. Results In 384 patients diagnosed with ES after haplo-HSCT, the median diagnostic time of ES was 11 days after haplo-HSCT, close to the time of neutrophil engraftment. Among them, 252 (65.6%) patients developed aGVHD, and 123 (32.0%) suffered II-IV aGVHD. The 3-year survival rate of ES patients who developed II-IV aGVHD was 17.1% (21/123) compared with 4.3% (11/252) in those without II-IV aGVHD (p&amp;lt;.001).Through multivariable logistic binary regression models, six prognosis risk factors were selected，including age at HSCT≤25 years(p=0.008; odds ratio [OR], 4.47; 95% confidence interval [CI], 1.54-14.30), obesity (BMI≥28) at HSCT(p=0.039; OR, 12.60; 95% CI, 1.30-194.14), previous pregnancy history (p=0.09; OR, 3.42; 95% CI, 0.83-18.45), time from diagnosis to transplantation≥6 months (p=0.027; OR, 2.73; 95% CI, 0.97-8.50), a high dose of MNC administration (≥9×10 8)(p=0.003; OR, 4.39; 95% CI, 1.70-12.41) and a high dose platelet transfusion (≥2u) during conditioning regimen (p=0.007; OR, 4.04; 95% CI, 1.51-11.73). According to the regression coefficient, 2 scores were assigned for obesity at HSCT, and 1 score was assigned for the other risk factors. Hence, a prognostic model for II-IV aGVHD termed TOPMAP ( time from diagnosis to transplantation, obesity, previous pregnancy history, high-dose MNC administration, age, and high-dose platelet transfusion) was constructed. Patients were stratified into 3 risk groups: low risk (0-2), intermediate risk (3-4), and high risk (5-7), which showed an incidence of II-IV aGVHD of 15.9% for low risk, 39.2% for intermediate risk, and 83.3% for high risk (P&amp;lt;0.001). The AUCs of the prognostic scoring model were 0.816 (95% CI 0.727-0.903) and 0.813 (0.759-0.868) for the internal validation and validation cohorts, respectively. The calibration plots showed a high agreement between the predicted and observed outcomes. Decision curve analysis indicated that patients with engraftment syndrome could benefit from the clinical application of the prognostic model. Conclusion Here, we reported the largest study on ES posthaplo-HSCT and found risk factors for ES onset. We developed the first straightforward scoring model (designated TOPMAP) that incorporates clinical and laboratory risk factors to predict the high-risk group for II-IV aGVHD in patients with ES after haplo-HSCT. This model can effectively help clinicians identify ES patients early, sectionalize them according to risk for II-IV aGVHD, carry out aGVHD prevention and improve patient outcomes.

Unsupervised Physics-Informed Health Indicator Estimation for Complex Systems

Developing Health Indicators (HI) is a crucial aspect of prognostics and health management of complex systems. Previous research has demonstrated the benefits of accurately determining the HI, which can lead to better performance of prognostic models. However, the existing methodologies for determining HI in complex systems are mostly semi-supervised and rely on assumptions that may not hold in real-world scenarios. The existing methods usually involve using a reference set of healthy sensor readings or run-to-failure data to infer HI. But the unsupervised inference of HI from sensor readings, which is challenging in scenarios where diverse operating conditions can mask the effect of degradation on sensor readings, has not been extensively researched. In this paper, we propose a novel physics-informed unsupervised model for determining HI. Unlike previous methods, constrained by assumptions, the proposed method uses prior general knowledge about degradation to infer HI, thereby eliminating the need for a reference set of healthy sensor readings. The proposed unsupervised model is an Autoencoder that incorporates constraints on its latent space to ensure consistency with knowledge about degradation. We assess the efficacy of the proposed model by analyzing a prevalent prognostic case study, specifically the turbofan engine dataset (N-CMAPSS). Our analysis considers the model's sensitivity to data availability and the resulting Health Index's quality, including trendability and monotonicity. Additionally, we investigate the impact of incorporating the Health Index in predicting Remaining Useful Life (RUL). We demonstrate that our proposed method generates a Health Index that exhibits greater monotonicity and trendability than the current state-of-the-art semi-supervised approach. Moreover, our approach for identifying the Health Index leads to enhanced prognostic performance compared to the existing semi-supervised approach.

Predicting 10-year breast cancer mortality risk in the general female population in England: a model development and validation study

Identifying female individuals at highest risk of developing life-threatening breast cancers could inform novel stratified early detection and prevention strategies to reduce breast cancer mortality, rather than only considering cancer incidence. We aimed to develop a prognostic model that accurately predicts the 10-year risk of breast cancer mortality in female individuals without breast cancer at baseline. In this model development and validation study, we used an open cohort study from the QResearch primary care database, which was linked to secondary care and national cancer and mortality registers in England, UK. The data extracted were from female individuals aged 20-90 years without previous breast cancer or ductal carcinoma in situ who entered the cohort between Jan 1, 2000, and Dec 31, 2020. The primary outcome was breast cancer-related death, which was assessed in the full dataset. Cox proportional hazards, competing risks regression, XGBoost, and neural network modelling approaches were used to predict the risk of breast cancer death within 10 years using routinely collected health-care data. Death due to causes other than breast cancer was the competing risk. Internal-external validation was used to evaluate prognostic model performance (using Harrell's C, calibration slope, and calibration in the large), performance heterogeneity, and transportability. Internal-external validation involved dataset partitioning by time period and geographical region. Decision curve analysis was used to assess clinical utility. We identified data for 11 626 969 female individuals, with 70 095 574 person-years of follow-up. There were 142 712 (1·2%) diagnoses of breast cancer, 24 043 (0·2%) breast cancer-related deaths, and 696 106 (6·0%) deaths from other causes. Meta-analysis pooled estimates of Harrell's C were highest for the competing risks model (0·932, 95% CI 0·917-0·946). The competing risks model was well calibrated overall (slope 1·011, 95% CI 0·978-1·044), and across different ethnic groups. Decision curve analysis suggested favourable clinical utility across all age groups. The XGBoost and neural network models had variable performance across age and ethnic groups. A model that predicts the combined risk of developing and then dying from breast cancer at the population level could inform stratified screening or chemoprevention strategies. Further evaluation of the competing risks model should comprise effect and health economic assessment of model-informed strategies. Cancer Research UK.

Clinical implication of the 2020 International Association for the Study of Lung Cancer histologic grading in surgically resected pathologic stage 1 lung adenocarcinomas: Prognostic value and association with computed tomography characteristics

ObjectivesTo investigate the incremental prognostic value of the 2020 International Association for the Study of Lung Cancer (IASLC) histologic grading system over traditional prognosticators in surgically resected pathologic stage 1 lung adenocarcinomas and to identify the clinical and radiologic characteristics of lung adenocarcinomas reclassified by the 2020 histologic grading system. Materials and methodsWe retrospectively enrolled 356 patients who underwent surgery for pathologic stage 1 adenocarcinoma between January 2016 and December 2017. The histologic grading was classified according to the predominant histologic subtype (conventional system) and the updated 2020 IASLC grading system. The clinical and computed tomography (CT) characteristics were compared according to the reclassification of the updated system. The performance of prognostic models for recurrence-free survival based on the combination of pathologic tumor size, histologic grade, and CT-based information was compared using the c-index. ResultsPostoperative recurrence occurred in 6.7% of patients during the follow-up period (mean, 1589.2 ± 406.7 days). Fifty-nine of 244 (24.2%) tumors with intermediate grades in the conventional system were reclassified as grade 3 with the updated grading system. They showed significantly larger solid proportions and higher percentages of pure solid nodules on CT compared to tumors without reclassification (n = 185) (P < 0.05). Prognostic prediction models based on pathology tumor size and histologic grades had significantly higher c-indices (0.754–0.803) compared to the model based on pathologic tumor size only (c-index:0.723, P < 0.05). ConclusionThe 2020 IASLC histologic grading system has significant incremental prognostic value over the pathologic stage in surgically resected pathologic stage 1 lung adenocarcinoma. Reclassified lung adenocarcinomas using the updated grading system have a larger solid proportion and a higher percentage of pure solid nodules on CT.

MR Intensity Normalization Methods Impact Sequence Specific Radiomics Prognostic Model Performance in Primary and Recurrent High-Grade Glioma.

This study investigates the impact of different intensity normalization (IN) methods on the overall survival (OS) radiomics models' performance of MR sequences in primary (pHGG) and recurrent high-grade glioma (rHGG). MR scans acquired before radiotherapy were retrieved from two independent cohorts (rHGG C1: 197, pHGG C2: 141) from multiple scanners (15, 14). The sequences are T1 weighted (w), contrast-enhanced T1w (T1wce), T2w, and T2w-FLAIR. Sequence-specific significant features (SF) associated with OS, extracted from the tumour volume, were derived after applying 15 different IN methods. Survival analyses were conducted using Cox proportional hazard (CPH) and Poisson regression (POI) models. A ranking score was assigned based on the 10-fold cross-validated (CV) concordance index (C-I), mean square error (MSE), and the Akaike information criterion (AICs), to evaluate the methods' performance. Scatter plots of the 10-CV C-I and MSE against the AIC showed an impact on the survival predictions between the IN methods and MR sequences (C1/C2 C-I range: 0.62-0.71/0.61-0.72, MSE range: 0.20-0.42/0.13-0.22). White stripe showed stable results for T1wce (C1/C2 C-I: 0.71/0.65, MSE: 0.21/0.14). Combat (0.68/0.62, 0.22/0.15) and histogram matching (HM, 0.67/0.64, 0.22/0.15) showed consistent prediction results for T2w models. They were also the top-performing methods for T1w in C2 (Combat: 0.67, 0.13; HM: 0.67, 0.13); however, only HM achieved high predictions in C1 (0.66, 0.22). After eliminating IN impacted SF using Spearman's rank-order correlation coefficient, a mean decrease in the C-I and MSE of 0.05 and 0.03 was observed in all four sequences. The IN method impacted the predictive power of survival models; thus, performance is sequence-dependent.

Resampling and harmonization for mitigation of heterogeneity in image parameters of baseline scans

Our study investigates the effects of heterogeneity in image parameters on the reproducibility of prognostic performance of models built using radiomic biomarkers. We compare the prognostic performance of models derived from the heterogeneity-mitigated features with that of models obtained from raw features, to assess whether reproducibility of prognostic scores improves upon application of our methods. We used two datasets: The Breast I-SPY1 dataset—Baseline DCE-MRI scans of 156 women with locally advanced breast cancer, treated with neoadjuvant chemotherapy, publicly available via The Cancer Imaging Archive (TCIA); The NSCLC IO dataset—Baseline CT scans of 107 patients with stage 4 non-small cell lung cancer (NSCLC), treated with pembrolizumab immunotherapy at our institution. Radiomic features (n = 102) are extracted from the tumor ROIs. We use a variety of resampling and harmonization scenarios to mitigate the heterogeneity in image parameters. The patients were divided into groups based on batch variables. For each group, the radiomic phenotypes are combined with the clinical covariates into a prognostic model. The performance of the groups is assessed using the c-statistic, derived from a Cox proportional hazards model fitted on all patients within a group. The heterogeneity-mitigation scenario (radiomic features, derived from images that have been resampled to minimum voxel spacing, are harmonized using the image acquisition parameters as batch variables) gave models with highest prognostic scores (for e.g., IO dataset; batch variable: high kernel resolution—c-score: 0.66). The prognostic performance of patient groups is not comparable in case of models built using non-heterogeneity mitigated features (for e.g., I-SPY1 dataset; batch variable: small pixel spacing—c-score: 0.54, large pixel spacing—c-score: 0.65). The prognostic performance of patient groups is closer in case of heterogeneity-mitigated scenarios (for e.g., scenario—harmonize by voxel spacing parameters: IO dataset; thin slice—c-score: 0.62, thick slice—c-score: 0.60). Our results indicate that accounting for heterogeneity in image parameters is important to obtain more reproducible prognostic scores, irrespective of image site or modality. For non-heterogeneity mitigated models, the prognostic scores are not comparable across patient groups divided based on batch variables. This study can be a step in the direction of constructing reproducible radiomic biomarkers, thus increasing their application in clinical decision making.

Prognostic Impact and Landscape of Mitochondrial DNA Variants in Myelodysplastic Syndromes Undergoing Allogenic Hematopoietic Cell Transplantation

Background: Myelodysplastic syndromes (MDS) are a heterogenous group of clonal hematopoietic cell disorders characterized by blood cytopenias and a tendency to progress to acute myeloid leukemia. Despite treatment advances, allogeneic hematopoietic stem-cell transplantation (allo-HCT) remains the only potentially curative therapy for MDS. However, mortality after allo-HCT is high due to disease relapse and transplant-related complications. Current prognostic tools defining MDS risk groups, such as the International Prognostic Scoring System (IPSS) and the revised IPSS (IPSS-R), are used to predict posttransplant outcomes for clinical decision-making. A major limitation of these scoring systems is a failure to account for variables other than clinical and hematologic parameters. Comprehensive genomic analyses have shown that mutations in specific genes (e.g., TP53, TET2, and DNMT3A) could inform prognostic stratification of MDS undergoing allo-HCT. However, most studies of MDS genome have focused on mutations occurring in the nuclear genome and ignored the mitochondrial genome. The current study aims to characterize mitochondrial genomic landscape of MDS and identify mitochondrial DNA (mtDNA) genetic variants prognostic for post allo-HCT outcomes in MDS. Methods: We performed whole genome sequencing (WGS) on whole-blood samples obtained before allo-HCT from 494 patients with MDS who were enrolled in the Center for International Blood and Marrow Transplant Research (CIBMTR) between 2014 and 2018. Mitochondrial genome sequences were extracted from the WGS data and aligned to the mtDNA reference genome (Revised Cambridge Reference Sequence, rCRS) by GSNAP. Putative mtDNA variants, including single nucleotide variants (SNV) and indels, were identified by parsing the SAM CIGAR string. Pathogenicity prediction was conducted for mtDNA variants in the protein-coding genes, MT-tRNAs and MT-rRNAs using different databases. We evaluated the impact of mtDNA variants on the overall survival (OS) and relapse with adjustment for MDS- and HCT-related factors. A random survival forest algorithm was applied to evaluate the prognostic performance of models that include mtDNA variants alone and combined with MDS- and HCT-related clinical factors. Results: The patients in the cohort have a median age of 67 years, 36.4% had poor/very poor Revised International Prognostic Scores (IPSS-R), and 26% received myeloablative conditioning regimens. On average, the sequencing depth of the mitochondrial genome was 14,596× (213× - 17,070×), and the average coverage was 99.90% (99.89% - 100%). A total of 2,666 mtDNA variants are identified, including 411 potential pathogenic variants. Mitochondrial control region (also known as D-Loop) was the most frequently mutated region in our cohort and the most frequently mutated protein-coding gene was MT-ND5 (16%). MDS patients with higher number of mitochondrial mutations (continuous variable) had significantly worse OS (hazard ratio, HR, 1.11; 95% CI, 1.01-1.22; P = 0.029) and higher risk of relapse (HR, 1.13; 95% CI, 1.00-1.27; P = 0.049) than patients with lower number of mutations. Gene-based analysis yielded significant associations with OS for MT-CYB (P = 1.04×10-3), MT-ND2 (P = 3.06×10-3) and MT-ND4 (P = 1.41×10-3) variants after Bonferroni correction (P < 0.05/16=3.13×10-3) (Table 1). Four mtDNA genes were significantly associated with relapse, including MT-CYB (P = 7.03×10-4), MT-ND2 (P = 5.92×10-4), MT-ND5 (P = 1.91×10-3) and MT-tRNA (P = 1.99×10-4). (Table 1). To investigate whether mtDNA variants could improve the prognostic stratification of MDS patients receiving allo-HCT, we fitted random survival forest models with and without inclusion of mtDNA variants in the models. The model based only on mtDNA genes had a c-index of 0.58 to predict OS, which was slightly higher than the IPSS-R (c-index = 0.48) and clinical model (IPSS-R plus MDS type and pre-transplantation treatments, c-index = 0.57). Adding mtDNA genes improved the predictive performance of the model, with c-index increased from 0.48 to 0.63 for the IPSS-R model and from 0.57 to 0.66 for the clinical model (Figure 1). Similar results were observed for the models in predicting relapse (Figure 1). Conclusions: There may be clinical utility of mtDNA variants to predict allo-HCT outcomes in combination with more standard clinical parameters. Figure 1View largeDownload PPTFigure 1View largeDownload PPT Close modal

Assessing the impact of MRI based diagnostics on pre-treatment disease classification and prognostic model performance in men diagnosed with new prostate cancer from an unscreened population

IntroductionPre-treatment risk and prognostic groups are the cornerstone for deciding management in non-metastatic prostate cancer. All however, were developed in the pre-MRI era. Here we compared categorisation of cancers using either only clinical parameters or with MRI enhanced information in men referred for suspected prostate cancer from an unscreened population.Patient and methodsData from men referred from primary care to our diagnostic service and with both clinical (digital rectal examination [DRE] and systematic biopsies) and MRI enhanced attributes (MRI stage and combined systematic/targeted biopsies) were used for this study. Clinical vs MRI data were contrasted for clinico-pathological and risk group re-distribution using the European Association of Urology (EAU), American Urological Association (AUA) and UK National Institute for Health Care Excellence (NICE) Cambridge Prognostic Group (CPG) models. Differences were retrofitted to a population cohort with long-term prostate cancer mortality (PCM) outcomes to simulate impact on model performance. We further contrasted individualised overall survival (OS) predictions using the Predict Prostate algorithm.ResultsData from 370 men were included (median age 66y). Pre-biopsy MRI stage reassignments occurred in 7.8% (versus DRE). Image-guided biopsies increased Grade Group 2 and ≥ Grade Group 3 assignments in 2.7% and 2.9% respectively. The main change in risk groups was more high-risk cancers (6.2% increase in the EAU and AUA system, 4.3% increase in CPG4 and 1.9% CPG5). When extrapolated to a historical population-based cohort (n = 10,139) the redistribution resulted in generally lower concordance indices for PCM. The 5-tier NICE-CPG system outperformed the 4-tier AUA and 3-tier EAU models (C Index 0.70 versus 0.65 and 0.64). Using an individualised prognostic model, changes in predicted OS were small (median difference 1% and 2% at 10- and 15-years’ respectively). Similarly, estimated treatment survival benefit changes were minimal (1% at both 10- and 15-years’ time frame).ConclusionMRI guided diagnostics does change pre-treatment risk groups assignments but the overall prognostic impact appears modest in men referred from unscreened populations. Particularly, when using more granular tiers or individualised prognostic models. Existing risk and prognostic models can continue to be used to counsel men about treatment option until long term survival outcomes are available.

Performance of prognostic models incorporating KRAS mutation status to predict survival after resection of colorectal liver metastases.

The Genetic And Morphologic Evaluation (GAME) score and modified clinical score (m-CS) are two novel prognostic models that incorporate KRAS mutation status to predict survival after resection of colorectal liver metastases (CRLM). This retrospective cohort study evaluated the performance of these two models. A total of 103 patients who underwent resection of CRLM between 2007 and 2017 and had known KRAS mutation status were included, 39 (37.9%) of whom had KRAS mutated tumours. Complete case analysis of the patients was performed according to the Clinical Risk Score (CRS), m-CS, and GAME score. The primary outcome was overall survival stratified according to low-risk and high-risk scores. Harrell's C-index and Akaike information criterion (AIC) were used to compare the discrimination of the evaluated prognostic models. The GAME score demonstrated the largest difference in overall survival for patients stratified according to low-risk and high-risk groups. Harrell's C-index values for the CRS, m-CS, and GAME models were 0.583, 0.600, and 0.668, respectively. AIC values for the CRS, m-CS, and GAME models were 441, 439, and 427, respectively. The GAME score outperforms the CRS and m-CS in predicting overall survival after resection of CRLM in patients with known KRAS mutation status.

Screening of Prognostic Markers for Hepatocellular Carcinoma Patients Based on Multichip Combined Analysis.

Methods GSE (14520, 36376, 57957, 76427) datasets were accessed from GEO database. 55 differential mRNAs (DEGs) were obtained by differential analysis based on the datasets. GO and KEGG analysis results indicated that the DEGs were enriched in xenobiotic metabolic process and other pathways. Expression profiles and clinical data of TCGA-LIHC mRNAs were from TCGA database. We established a prognostic model of HCC through univariate and multivariate Cox risk regression analyses. ROC curve analysis was used to examine the prognostic model performance. GSEA analysis was performed between the high- and low-risk score sample groups. Results A 4-gene HCC prognostic model was constructed, in which the gene expressions correlated to HCC patients' survival. The AUC value presented 0.734 in the ROC analysis for the prognostic model. Conclusion The four-gene model could be introduced as an independent prognostic factors to assess HCC patients' survival status.

A Novel Necroptosis-Related Gene Signature in Skin Cutaneous Melanoma Prognosis and Tumor Microenvironment.

Background: Necroptosis has been identified recently as a newly recognized programmed cell death that has an impact on tumor progression and prognosis, although the necroptosis-related gene (NRGs) potential prognostic value in skin cutaneous melanoma (SKCM) has not been identified. The aim of this study was to construct a prognostic model of SKCM through NRGs in order to help SKCM patients obtain precise clinical treatment strategies. Methods: RNA sequencing data collected from The Cancer Genome Atlas (TCGA) were used to identify differentially expressed and prognostic NRGs in SKCM. Depending on 10 NRGs via the univariate Cox regression analysis usage and LASSO algorithm, the prognostic risk model had been built. It was further validated by the Gene Expression Omnibus (GEO) database. The prognostic model performance had been assessed using receiver operating characteristic (ROC) curves. We evaluated the predictive power of the prognostic model for tumor microenvironment (TME) and immunotherapy response. Results: We constructed a prognostic model based on 10 NRGs (FASLG, TLR3, ZBP1, TNFRSF1B, USP22, PLK1, GATA3, EGFR, TARDBP, and TNFRSF21) and classified patients into two high- and low-risk groups based on risk scores. The risk score was considered a predictive factor in the two risk groups regarding the Cox regression analysis. A predictive nomogram had been built for providing a more beneficial prognostic indicator for the clinic. Functional enrichment analysis showed significant enrichment of immune-related signaling pathways, a higher degree of immune cell infiltration in the low-risk group than in the high-risk group, a negative correlation between risk scores and most immune checkpoint inhibitors (ICIs), anticancer immunity steps, and a more sensitive response to immunotherapy in the low-risk group. Conclusions: This risk score signature could be applied to assess the prognosis and classify low- and high-risk SKCM patients and help make the immunotherapeutic strategy decision.

Learning patient-level prediction models across multiple healthcare databases: evaluation of ensembles for increasing model transportability

BackgroundPrognostic models that are accurate could help aid medical decision making. Large observational databases often contain temporal medical data for large and diverse populations of patients. It may be possible to learn prognostic models using the large observational data. Often the performance of a prognostic model undesirably worsens when transported to a different database (or into a clinical setting). In this study we investigate different ensemble approaches that combine prognostic models independently developed using different databases (a simple federated learning approach) to determine whether ensembles that combine models developed across databases can improve model transportability (perform better in new data than single database models)?MethodsFor a given prediction question we independently trained five single database models each using a different observational healthcare database. We then developed and investigated numerous ensemble models (fusion, stacking and mixture of experts) that combined the different database models. Performance of each model was investigated via discrimination and calibration using a leave one dataset out technique, i.e., hold out one database to use for validation and use the remaining four datasets for model development. The internal validation of a model developed using the hold out database was calculated and presented as the ‘internal benchmark’ for comparison.ResultsIn this study the fusion ensembles generally outperformed the single database models when transported to a previously unseen database and the performances were more consistent across unseen databases. Stacking ensembles performed poorly in terms of discrimination when the labels in the unseen database were limited. Calibration was consistently poor when both ensembles and single database models were applied to previously unseen databases.ConclusionA simple federated learning approach that implements ensemble techniques to combine models independently developed across different databases for the same prediction question may improve the discriminative performance in new data (new database or clinical setting) but will need to be recalibrated using the new data. This could help medical decision making by improving prognostic model performance.

Identifying prognostic factors for clinical outcomes and costs in four high-volume surgical treatments using routinely collected hospital data

Identifying prognostic factors (PFs) is often costly and labor-intensive. Routinely collected hospital data provide opportunities to identify clinically relevant PFs and construct accurate prognostic models without additional data-collection costs. This multicenter (66 hospitals) study reports on associations various patient-level variables have with outcomes and costs. Outcomes were in-hospital mortality, intensive care unit (ICU) admission, length of stay, 30-day readmission, 30-day reintervention and in-hospital costs. Candidate PFs were age, sex, Elixhauser Comorbidity Score, prior hospitalizations, prior days spent in hospital, and socio-economic status. Included patients dealt with either colorectal carcinoma (CRC, n = 10,254), urinary bladder carcinoma (UBC, n = 17,385), acute percutaneous coronary intervention (aPCI, n = 25,818), or total knee arthroplasty (TKA, n = 39,214). Prior hospitalization significantly increased readmission risk in all treatments (OR between 2.15 and 25.50), whereas prior days spent in hospital decreased this risk (OR between 0.55 and 0.95). In CRC patients, women had lower risk of in-hospital mortality (OR 0.64), ICU admittance (OR 0.68) and 30-day reintervention (OR 0.70). Prior hospitalization was the strongest PF for higher costs across all treatments (31–64% costs increase/hospitalization). Prognostic model performance (c-statistic) ranged 0.67–0.92, with Brier scores below 0.08. R-squared ranged from 0.06–0.19 for LoS and 0.19–0.38 for costs. Identified PFs should be considered as building blocks for treatment-specific prognostic models and information for monitoring patients after surgery. Researchers and clinicians might benefit from gaining a better insight into the drivers behind (costs) prognosis.