Performance Of Risk Prediction Models Research Articles

Risk prediction models for incident heart failure: a systematic review and meta-analysis

Abstract Background Heart failure (HF) risk prediction models combine multivariable patient data to estimate an individual's risk of developing HF. By detecting at-risk and early-stage patients, models may facilitate earlier intervention to prevent or delay HF development. Previous systematic reviews were unable to recommend any existing prediction models for clinical use due to insufficient evidence and lack of guidelines on appraising study quality at their time of publication. Purpose To summarize the performance of risk prediction models for incident HF and identify models for further validation and potential clinical use. Methods We searched MEDLINE and EMBASE in June 2021 for English-language studies developing or validating HF risk prediction models. Studies were also retrieved from two previous systematic reviews. We narratively summarized model characteristics (e.g. model type, predictors used, prediction horizon) and study methodology (e.g. validation methods). Performance was assessed among all models validated in ≥ 1 cohort. For all models validated in ≥ 2 cohorts, we pooled discrimination measures using random-effects meta-analyses. Calibration was descriptively summarized based on individual study results from statistical tests and graph digitization of calibration plots. Study quality was assessed using the Prediction model Risk Of Bias ASsessment Tool (PROBAST). Results Of 18,937 publications screened, 41 studies consisting of 120 prediction models were included. Twenty models were both derived and validated, 99 only derived, and 1 only validated. Risk of bias was rated as high in nearly all (94.7%) PROBAST assessments, mostly attributable to issues with analysis. Among 21 models validated in ≥ 1 cohort, most had moderate (61.9%, C-statistic 0.7 to <0.8) or high (23.8%, C-statistic 0.8 to <0.9) discrimination. In patients with low predicted risk (<10%), the calibration was adequate. Nine (42.9%) of these 21 models were presented as web-based calculators and five (23.8%) as points-based risk scores. Based on performance, number of validation cohorts, study risk of bias, and user friendliness, the Atherosclerosis Risk in Communities (ARIC), Multi-Ethnic Study of Atherosclerosis (MESA), Pooled Cohort equations to Prevent Heart Failure (PCP-HF), and Health ABC models emerged as the most promising risk scores for clinical practice. Conclusions Given their acceptable performance but high risk of bias, future studies should focus on the external validation of these models in studies of high methodological rigor. Models should be validated in a greater diversity of patient populations, particularly with respect to race. Impact analyses assessing how the clinical implementation of these models affects patient outcomes are also required prior to their routine use. Once validated, these models may help guide clinical decision making to prevent the onset of HF with early, aggressive risk factor modification.Discrimination in 21 validated modelsCharacteristics of recommended models

Performance of risk prediction models for diabetic foot ulcer: a meta-analysis.

The number of prediction models for diabetic foot ulcer (DFU) risk is increasing, but their methodological quality and clinical applicability are uncertain. We conducted a systematic review to assess their performance. We searched PubMed, Cochrane Library, and Embase databases up to 10 February 2024 and extracted relevant information from selected prediction models. The Prediction model Risk Of Bias ASsessment Tool (PROBAST) checklist was used to assess bias risk and applicability. All statistical analyses were conducted in Stata 14.0. Initially, 13,562 studies were retrieved, leading to the inclusion of five development and five validation models from eight studies. DFU incidence ranged from 6% to 16.8%, with age and hemoglobin A1C (HbA1c) commonly used as predictive factors. All included studies had a high risk of bias, mainly due to disparities in population characteristics and methodology. In the meta-analysis, we observed area under the curve (AUC) values of 0.78 (95% CI [0.69-0.89]) for development models and 0.84 (95% CI [0.79-0.90]) for validation models. DFU risk prediction models show good overall accuracy, but there is a risk of bias. Adherence to the PROBAST checklist is crucial for improving their clinical applicability.

Abstract PO3-08-01: Decision curve analysis to compare breast cancer risk predictions for a polygenic integrated clinical risk model with those of a gold standard

Abstract Risk-based guidelines have been developed to provide advice on options that are available to women who are at increased risk of breast cancer. We and others are focused on improving risk prediction models for general population use. For breast cancer, polygenic risk has been identified as important for risk prediction, especially when combined with clinical risk factors such as family history, anthropometric measures, breast imaging measures, and hormonal and reproductive risk factors. When comparing the performance of risk prediction models, the AUC is most often used but it focuses on performance in terms of distinguishing between affected and unaffected individuals and does not take into account differences in the specificity (true negative rate) and sensitivity (true positive rate) of the models being compared. In contrast, decision curve analysis takes sensitivity and specificity into account and enables the comparison of the net benefit of risk prediction tools at the clinical risk thresholds that are used to guide clinical management decisions. Herein, we use decision curve analysis to compare breast cancer risk prediction guidelines for BRISK and the Breast Cancer Risk Assessment Tool (BCRAT) in the UK Biobank, and for BRISK and IBIS version 7 in the Nurses’ Health Study. We evaluated the net benefit at the 5-year risk thresholds (1.67% and 3%) that are used to guide clinical management decisions around risk-reducing medication and at the remaining lifetime risk of 20% that is used to guide supplemental MRI surveillance. In the UK Biobank, BRISK showed a net benefit over BCRAT at the 5-year risk thresholds of 1.67% and 3.0%. The AUCs were 0.649 (95% CI = 0.640, 0.695) for BRISK and 0.567 (95% CI = 0.556, 0.577) for BCRAT and showed that, overall, BRISK discriminated between affected and unaffected women better than BCRAT (P < 0.001). In the Nurses’ Health Study, BRISK showed a net benefit over IBIS at the remaining lifetime risk threshold of 20%. The AUCs were 0.647; 95% CI = 0.627, 0.668 for BRISK and 0.571; 95% CI = 0.546, 0.595 for IBIS, a statistically significant improvement for BRISK over IBIS (P < 0.0001). We also looked at sensitivity, specificity, positive predictive value and negative predictive value even though these risk models are one-step removed from breast cancer diagnosis by screening/risk-reduction options. The positive predictive values are higher for the BRISK (27.1%) model compared to BCRAT (5.5%) in the UK Biobank dataset and higher for BRISK (15.4%) compared to IBISv7 (14.0%) in the Nurses’ dataset. We have shown the application of a statistical tool that can be used to directly compare risk models at clinically relevant thresholds. We have shown that BRISK, which comprises polygenic risk and clinical risk factors, shows higher net benefit at both 1.67% and 3% thresholds compared to BCRAT and at 20% remaining lifetime risk for IBIS version 7. In breast cancer risk prediction, a tool with high specificity (and therefore a necessary trade-off for low sensitivity) is preferred because we do not want to incorrectly classify women as not being at high risk and deny them the opportunity for intervention in the form of more frequent screening, alternative modes of screening or risk-reducing medication. This improvement in general population risk stratification that BRISK can provide has the potential to have a clinically significant effect on women’s health. Citation Format: Erika Spaeth, Bernard A Rosner, Gill Dite. Decision curve analysis to compare breast cancer risk predictions for a polygenic integrated clinical risk model with those of a gold standard [abstract]. In: Proceedings of the 2023 San Antonio Breast Cancer Symposium; 2023 Dec 5-9; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2024;84(9 Suppl):Abstract nr PO3-08-01.

Performance of risk prediction models for post-liver transplant patient and graft survival over time.

Given liver transplantation organ scarcity, selection of recipients and donors to maximize post-transplant benefit is paramount. Several scores predict post-transplant outcomes by isolating elements of donor and recipient risk, including the donor risk index, Balance of Risk, pre-allocation score to predict survival outcomes following liver transplantation/survival outcomes following liver transplantation (SOFT), improved donor-to-recipient allocation score for deceased donors only/improved donor-to-recipient allocation score for both deceased and living donors (ID2EAL-D/-DR), and survival benefit (SB) models. No studies have examined the performance of these models over time, which is critical in an ever-evolving transplant landscape. This was a retrospective cohort study of liver transplantation events in the UNOS database from 2002 to 2021. We used Cox regression to evaluate model discrimination (Harrell's C) and calibration (testing of calibration curves) for post-transplant patient and graft survival at specified post-transplant timepoints. Sub-analyses were performed in the modern transplant era (post-2014) and for key donor-recipient characteristics. A total of 112,357 transplants were included. The SB and SOFT scores had the highest discrimination for short-term patient and graft survival, including in the modern transplant era, where only the SB model had good discrimination (C ≥ 0.60) for all patient and graft outcome timepoints. However, these models had evidence of poor calibration at 3- and 5-year patient survival timepoints. The ID2EAL-DR score had lower discrimination but adequate calibration at all patient survival timepoints. In stratified analyses, SB and SOFT scores performed better in younger (< 40 y) and higher Model for End-Stage Liver Disease (≥ 25) patients. All prediction scores had declining discrimination over time, and scores relying on donor factors alone had poor performance. Although the SB and SOFT scores had the best overall performance, all models demonstrated declining performance over time. This underscores the importance of periodically updating and/or developing new prediction models to reflect the evolving transplant field. Scores relying on donor factors alone do not meaningfully inform post-transplant risk.

A comparison of hyperparameter tuning procedures for clinical prediction models: A simulation study.

Tuning hyperparameters, such as the regularization parameter in Ridge or Lasso regression, is often aimed at improving the predictive performance of risk prediction models. In this study, various hyperparameter tuning procedures for clinical prediction models were systematically compared and evaluated in low-dimensional data. The focus was on out-of-sample predictive performance (discrimination, calibration, and overall prediction error) of risk prediction models developed using Ridge, Lasso, Elastic Net, or Random Forest. The influence of sample size, number of predictors and events fraction on performance of the hyperparameter tuning procedures was studied using extensive simulations. The results indicate important differences between tuning procedures in calibration performance, while generally showing similar discriminative performance. The one-standard-error rule for tuning applied to cross-validation (1SE CV) often resulted in severe miscalibration. Standard non-repeated and repeated cross-validation (both 5-fold and 10-fold) performed similarly well and outperformed the other tuning procedures. Bootstrap showed a slight tendency to more severe miscalibration than standard cross-validation-based tuning procedures. Differences between tuning procedures were larger for smaller sample sizes, lower events fractions and fewer predictors. These results imply that the choice of tuning procedure can have a profound influence on the predictive performance of prediction models. The results support the application of standard 5-fold or 10-fold cross-validation that minimizes out-of-sample prediction error. Despite an increased computational burden, we found no clear benefit of repeated over non-repeated cross-validation for hyperparameter tuning. We warn against the potentially detrimental effects on model calibration of the popular 1SE CV rule for tuning prediction models in low-dimensional settings.

Bleeding Risk Prediction in Patients Treated with Antithrombotic Drugs According to the Anatomic Site of Bleeding, Indication for Treatment, and Time Since Treatment Initiation.

Background Reasons for the relatively poor performance of bleeding prediction models are not well understood but may relate to differences in predictors for various anatomical sites of bleeding. Methods We pooled individual participant data from four randomized controlled trials of antithrombotic therapy in patients with coronary and peripheral artery diseases, embolic stroke of undetermined source (ESUS), or atrial fibrillation. We examined discrimination and calibration of models for any major bleeding, major gastrointestinal (GI) bleeding, and intracranial hemorrhage (ICH), according to the time since initiation of antithrombotic therapy, and indication for antithrombotic therapy. Results Of 57,813 patients included, 1,948 (3.37%) experienced major bleeding, including 717 (1.24%) major GI bleeding and 274 (0.47%) ICH. The model derived to predict major bleeding at 1 year from any site (c-index, 0.69, 95% confidence interval [CI], 0.68-0.71) performed similarly when applied to predict major GI bleeding (0.71, 0.69-0.74), but less well to predict ICH (0.64, 0.61-0.69). Models derived to predict GI bleeding (0.75, 0.74-0.78) and ICH (0.72, 0.70-0.79) performed better than the general major bleeding model. Discrimination declined over time since the initiation of antithrombotic treatment, stabilizing at approximately 2 years for any major bleeding and major GI bleeding and 1 year for ICH. Discrimination was best for the model predicting ICH in the ESUS population (0.82, 0.78-0.92) and worst for the model predicting any major bleeding in the coronary and peripheral artery disease population (0.66, 0.65-0.69). Conclusion Performance of risk prediction models for major bleeding is affected by site of bleeding, time since initiation of antithrombotic therapy, and indication for antithrombotic therapy.

Application of deep neural survival networks to the development of risk prediction models for diabetes mellitus, hypertension, and dyslipidemia.

: Although numerous risk prediction models have been proposed, few such models have been developed using neural network-based survival analysis. We developed risk prediction models for three cardiovascular disease risk factors (diabetes mellitus, hypertension, and dyslipidemia) among a working-age population in Japan using DeepSurv, a deep feed-forward neural network. : Data were obtained from the Japan Epidemiology Collaboration on Occupational Health Study. A total of 51 258, 44 197, and 31 452 individuals were included in the development of risk models for diabetes mellitus, hypertension, and dyslipidemia, respectively; two-thirds of whom were used to develop prediction models, and the rest were used to validate the models. We compared the performances of DeepSurv-based models with those of prediction models based on the Cox proportional hazards model. : The area under the receiver-operating characteristic curve was 0.878 [95% confidence interval (CI) = 0.864-0.892] for diabetes mellitus, 0.835 (95% CI = 0.826-0.845) for hypertension, and 0.826 (95% CI = 0.817-0.835) for dyslipidemia. Compared with the Cox proportional hazards-based models, the DeepSurv-based models had better reclassification performance [diabetes mellitus: net reclassification improvement (NRI) = 0.474, P ≤ 0.001; hypertension: NRI = 0.194, P ≤ 0.001; dyslipidemia: NRI = 0.397, P ≤ 0.001] and discrimination performance [diabetes mellitus: integrated discrimination improvement (IDI) = 0.013, P ≤ 0.001; hypertension: IDI = 0.007, P ≤ 0.001; and dyslipidemia: IDI = 0.043, P ≤ 0.001]. : This study suggests that DeepSurv has the potential to improve the performance of risk prediction models for cardiovascular disease risk factors.

MA19.05 Performance of Risk Prediction Models to Select Individuals for Lung Cancer Screening in the European Population

MA19.05 Performance of Risk Prediction Models to Select Individuals for Lung Cancer Screening in the European Population

Comparison of likelihood penalization and variance decomposition approaches for clinical prediction models: A simulation study.

Logistic regression is one of the most commonly used approaches to develop clinical risk prediction models. Developers of such models often rely on approaches that aim to minimize the risk of overfitting and improve predictive performance of the logistic model, such as through likelihood penalization and variance decomposition techniques. We present an extensive simulation study that compares the out-of-sample predictive performance of risk prediction models derived using the elastic net, with Lasso and ridge as special cases, and variance decomposition techniques, namely, incomplete principal component regression and incomplete partial least squares regression. We varied the expected events per variable, event fraction, number of candidate predictors, presence of noise predictors, and the presence of sparse predictors in a full-factorial design. Predictive performance was compared on measures of discrimination, calibration, and prediction error. Simulation metamodels were derived to explain the performance differences within model derivation approaches. Our results indicate that, on average, prediction models developed using penalization and variance decomposition approaches outperform models developed using ordinary maximum likelihood estimation, with penalization approaches being consistently superior over the variance decomposition approaches. Differences in performance were most pronounced on the calibration of the model. Performance differences regarding prediction error and concordance statistic outcomes were often small between approaches. The use of likelihood penalization and variance decomposition techniques methods was illustrated in the context of peripheral arterialdisease.

Magnetic resonance imaging-radiomics in endometrial cancer: a systematic review and meta-analysis

ObjectiveEndometrial carcinoma is the most common gynecological tumor in developed countries. Clinicopathological factors and molecular subtypes are used to stratify the risk of recurrence and to tailor adjuvant treatment. The...

Abstract P341: A Flexible, Interpretable Approach to Recalibrate Risk Prediction Models: Adapting the Pooled Cohorts Equation to a Local New England Contemporary Population

Introduction: Atherosclerotic cardiovascular disease (ASCVD) prognostic models have well-documented poor generalizability in contemporary populations. Sustained, accurate performance of risk prediction models is challenging given demographic shifts, changes in practice patterns, and geographic variation. We propose an approach to tailor known risk prediction models for local settings while preserving known disease risk mechanisms. Here, as a case study, we recalibrate the Pooled Cohort Equations (PCE) in a contemporary (1997-2018), local electronic health records (EHR) cohort in New England. Methods: We conducted a retrospective cohort study using EHR from large Boston-area hospitals. 61,808 patients 45+ years old, without ASCVD in 1997-2006 and complete data to calculate PCE model were split into training (50%), validation (10%), and test sets (40%). We used XGBoost to recalibrate PCE model (XGB-PCE) using PCE risk score, age, race/ethnicity, and sex (Figure). We constrained the XGB-PCE model to preserve known physiologic trends i.e. increase in ASCVD risk with increase in total cholesterol and systolic blood pressure. Models’ discrimination and calibration were calculated in a set aside test set using the Harrell’s c-index and modified Nam D’Agostino χ2 test. Results: In the test set, XGB-PCE exhibited improved calibration across ASCVD risk categories (p=0.37) and race/ethnicity populations (p=0.73). In contrast, even after recalibration in the large, PCE was poorly calibrated across risk categories (p&lt;0.001) and race/ethnicity populations (p&lt;0.001). For model discrimination, both models produced similar c-index (Δc-index of ±.01). Conclusions: Our recalibration approach maintained comparable discriminative performance and demonstrated improved calibration relative to the PCE while preserving known disease risk mechanisms. Methods that are flexible and preserve disease mechanisms have the potential to locally tailor existing models to be better calibrated for contemporary populations.

Evaluating Dynamic Discrimination Performance of Risk Prediction Models for Survival Outcomes.

Risk prediction models for survival outcomes are widely applied in medical research to predict future risk for the occurrence of the event. In many clinical studies, the biomarker data are measured repeatedly over time. To facilitate timely disease prognosis and decision making, many dynamic prediction models have been developed and generate predictions on a real-time basis. As a dynamic prediction model updates an individual's risk prediction over time based on new measurements, it is often important to examine how well the model performs at different measurement times and prediction times. In this article, we propose a two-dimensional area under curve (AUC) measure for dynamic prediction models and develop associated estimation and inference procedures. The estimation procedures are discussed under two types of biomarker measurement schedules: regular visits and irregular visits. The model parameters are estimated effectively by maximizing a pseudo-partial likelihood function. We apply the proposed method to a renal transplantation study to evaluate the discrimination performance of dynamic prediction models based on longitudinal biomarkers for graft failure.

Performance of risk prediction models in breast cancer screening among women in Cyprus.

Tools that can predict breast cancer risk in women will have a significant impact on women and healthcare systems in low- and middle-income countries. We compared the performances of the Breast and Ovarian Analysis of Disease Incidence and Carrier Estimation Algorithm (BOADICEA) model, the International Breast Cancer Intervention Study (IBIS) model, and the Gail model in predicting breast cancer risk among women in Cyprus. We recruited 655 women from Dr Burhan Nalbantoglu Devlet Hastanesi Hospital in Lefkosa: 318 had breast cancer and 337 did not have breast cancer (hospital-based controls). We collected retrospective data from the hospital's medical records and interviews with the women after informed consent. Data collected included: age, age at diagnosis, age at menarche, menopausal status, presence of benign breast disease, breast cancer in relatives, BRCA-1 and BRCA-2 mutations, history of hormone replacement therapy, and breast density. We calculated the 5-year risk of breast cancer and the risk values were used to plot receiver operating curves. For the area under the curve (95% confidence interval, CI), sensitivity and specificity for the models were: BOADICEA 0.81 (95% CI: 0.77-0.84), 26.4% and 98.8%; IBIS 0.80 (95% CI: 0.77-0.84), 19.4% and 97.3%; and Gail 0.76 (95% CI: 0.73-0.80), 17.3% and 98.5%. The breast cancer risk prediction models performed similarly although on closer appraisal, the BOADICEA and IBIS models performed slightly better. These models are simple, appropriate, cost-effective, and non-invasive tools for identifying high-risk women in low- and middle-income countries who could benefit from mammography screening.

Risk prediction models in patients undergoing percutaneous coronary intervention: A collaborative analysis from a Japanese administrative dataset and nationwide academic procedure registry

BackgroundContemporary guidelines emphasize the importance of risk stratification in improving the quality of care for patients undergoing percutaneous coronary intervention (PCI). We aimed to investigate whether adding information from a procedure-based academic registry to administrative claims data would improve the performance of risk prediction model. MethodsWe combined two nationally representative administrative and clinical databases. The study cohort comprised 43,095 patients; 18,719 and 23, 525 with acute [ACS] and chronic [CCS] coronary syndrome, respectively. Each population was randomly divided into the logistic regression model (derivation cohort, 80%) and model validation (validation cohort, 20%) groups. The performances of the following models were compared using C-statistics: (1) variables restricted to baseline claims data (model #1), (2) clinical registry data (model #2), and (3) expanded to both claims and clinical registry data (model #3). The primary outcomes were in-hospital mortality and bleeding. ResultsThe primary outcomes occurred in 3.7% (in-hospital mortality)/5.0% (bleeding) of patients with ACS and 0.21%/0.95% of CCS patients. For each event, the model performance was 0.65 (95% confidence interval [CI], 0.60–0.69) /0.67 (0.63–0.71) in ACS and 0.52 (0.35–0.76) /0.62 (0.54–0.70) for CCS patients in model #1, 0.83 (0.80–0.87) /0.77 (0.74–0.81) in ACS and 0.76 (0.60–0.92) /0.67 (0.59–0.75) in CCS for model #2, and 0.83 (0.79–0.86) /0.78 (0.75–0.81) in ACS and 0.76 (0.61–0.92) /0.67 (0.58–0.74) in CCS for model #3. ConclusionsCombining clinical information from the academic registry with claims databases improved its performance in predicting adverse events.

Clinical Performance of the Consensus Immunoscore in Colon Cancer in the Asian Population from the Multicenter International SITC Study.

Simple SummaryResearch interest in Immuno-oncology and the role of the adaptative immune system in the progression and prognosis of colon cancer (CC) is growing. In this study, we evaluated the prognostic value of the consensus Immunoscore in 423 patients with AJCC/UICC-TNM stages I–III CC from Asian care centers. Immunoscore (IS) is a bench-to-digital pathology assay that quantifies CD3+ and cytotoxic CD8+ T-lymphocyte densities within the tumor and its invasive margin, stratifying patients into three categories: Low IS, Intermediate IS, and High IS. Multivariable Cox models stratified by center were used to assess the associations between Immunoscore and outcomes, adjusting for potential confounders, including gender, T-stage, N-stage, sidedness, and MSI. A comparison of the performance of risk prediction models was performed using the likelihood ratio test p-value. In uni/multivariable analyses, a High Immunoscore was significantly associated with prolonged survival of CC patients within the Asian population.BACKGROUND: In this study, we evaluated the prognostic value of Immunoscore in patients with stage I–III colon cancer (CC) in the Asian population. These patients were originally included in an international study led by the Society for Immunotherapy of Cancer (SITC) on 2681 patients with AJCC/UICC-TNM stages I–III CC. METHODS: CD3+ and cytotoxic CD8+ T-lymphocyte densities were quantified in the tumor and invasive margin by digital pathology. The association of Immunoscore with prognosis was evaluated for time to recurrence (TTR), disease-free survival (DFS), and overall survival (OS). RESULTS: Immunoscore stratified Asian patients (n = 423) into different risk categories and was not impacted by age. Recurrence-free rates at 3 years were 78.5%, 85.2%, and 98.3% for a Low, Intermediate, and High Immunoscore, respectively (HR[Low-vs-High] = 7.26 (95% CI 1.75−30.19); p = 0.0064). A High Immunoscore showed a significant association with prolonged TTR, OS, and DFS (p < 0.05). In Cox multivariable analysis stratified by center, Immunoscore association with TTR was independent (HR[Low-vs-Int+High] = 2.22 (95% CI 1.10–4.55) p = 0.0269) of the patient’s gender, T-stage, N-stage, sidedness, and MSI status. A significant association of a High Immunoscore with prolonged TTR was also found among MSS (HR[Low-vs-Int+High] = 4.58 (95% CI 2.27−9.23); p ≤ 0.0001), stage II (HR[Low-vs-Int+High] = 2.72 (95% CI 1.35−5.51); p = 0.0052), low-risk stage-II (HR[Low-vs-Int+High] = 2.62 (95% CI 1.21−5.68); p = 0.0146), and high-risk stage II patients (HR[Low-vs-Int+High] = 3.11 (95% CI 1.39−6.91); p = 0.0055). CONCLUSION: A High Immunoscore is significantly associated with the prolonged survival of CC patients within the Asian population.

Comparison between Carotid Distensibility-Based Vascular Age and Risk-Based Vascular Age in Middle-Aged Population Free of Cardiovascular Disease.

The concept of vascular age (VA) was proposed to provide patients with an understandable explanation of cardiovascular (CV) risk and to improve the performance of prediction models. The present study compared risk-based VA derived from Framingham Risk Score (FRS) and Systematic Coronary Risk Estimation (SCORE) models with value-based VA derived from the measurement of the common carotid artery (CCA) distensibility coefficient (DC), and it assessed the impact of DC-based VA on risk reclassification. In 528 middle-aged individuals apparently free of CV disease, DC was measured by radiofrequency-based arterial wall tracking that was previously utilised to establish sex- and age-specific reference values in a healthy population. DC-based VA represented the median value (50th percentile) for given sex in the reference population. FRS-based and SCORE-based VA was calculated as recommended. We observed a good agreement between DC-based and FRS-based VA, with a mean difference of 0.46 ± 12.2 years (p = 0.29), while the mean difference between DC-based and SCORE-based VA was higher (3.07 ± 12.7 years, p < 0.0001). When only nondiabetic individuals free of antihypertensive therapy were considered (n = 341), the mean difference dropped to 0.70 ± 12.8 years (p = 0.24). Substitution of chronological age with DC-based VA in FRS and SCORE models led to a reclassification of 28% and 49% of individuals, respectively, to the higher risk category. Our data suggest that the SCORE prediction model, in which diabetes and antihypertensive treatment are not considered, should be used as a screening tool only in healthy individuals. The use of VA derived from CCA distensibility measurements could improve the performance of risk prediction models, even that of the FRS model, as it might integrate risk prediction with additional risk factors participating in vascular ageing, unique to each individual. Prospective studies are needed to validate the role of DC-based VA in risk prediction.

Performance of risk prediction models for post-operative mortality in patients undergoing liver resection

BackgroundLiver resection is commonly performed for hepatic tumors, however preoperative risk stratification remains challenging. We evaluated the performance of contemporary prediction models for short-term mortality after liver resection in patients with and without cirrhosis. MethodsThis retrospective cohort study examined National Surgical Quality Improvement Program data. We included patients who underwent liver resections from 2014 to 2019. VOCAL-Penn, MELD, MELD-Na, ALBI, and Mayo risk scores were evaluated in terms of model discrimination and calibration for 30-day post-operative mortality. ResultsA total 15,198 patients underwent liver resection, of whom 249 (1.6%) experienced 30-day post-operative mortality. The VOCAL-Penn score had the highest discrimination (area under the ROC curve [AUC] 0.74) compared to all other models. The VOCAL-Penn score similarly outperformed other models in patients with (AUC 0.70) and without (AUC 0.74) cirrhosis. ConclusionThe VOCAL-Penn score demonstrated superior predictive performance for 30-day post-operative mortality after liver resection as compared to existing clinical standards.

Immune-Related Biomarkers Improve Performance of Risk Prediction Models for Survival in Patients With Hepatocellular Carcinoma.

ObjectThe prediction of hepatocellular carcinoma (HCC) prognosis faced great challenge due to tumor heterogeneity. The purpose of this study was to explore the correlation between the immune infiltrate and prognosis. Moreover, we aimed to establish a risk prediction model for survival in HCC patients based on clinicopathological and immune indicators.MethodsIn this study, 316 patients with HCC who underwent radical resection in West China Hospital from 2009 to 2014 were included. Clinicopathological data and pathological specimens were collected. H&E staining and immunohistochemical staining were performed on the pathological tissue sections. The evaluation of tumor-infiltrating lymphocyte (TIL) density was based on H&E slices, and the assessment of the expressions of CD8, CD68, Lymphocyte activation gene-3 (LAG-3), T cell immunoglobulin domain and mucin domain-3 (TIM-3), Programmed Cell Death Protein 1 (PD-1), Programmed Cell Death Ligand 1 (PD-L1), OX40, CD66b, and Tryptase. was performed on the immunohistochemical slices. A risk prediction model for survival in HCC patients was established by integrating immune-related biomarkers and clinicopathological indicators.ResultsThe Barcelona Clinic Liver Cancer (BCLC) stage; the microvascular invasion status; the density of TILs; the expressing levels of CD66b, OX40, and PD-L1 in the immune cell; CD68; and CD8 were the predictors of patients’ overall survival (OS). The BCLC stage; the density of TILs; and the expressions of OX40, CD68, and CD8 were associated with disease-free survival (DFS). The expressions of CD66b, CD68, OX40, and CD8 had a cumulative effect on prognosis. The area under the curve of the prediction model for OS based on clinicopathological features was improved from 0.62 to 0.74 by adding to CD8, OX40, CD68, CD66b, and TILs, whereas it was improved from 0.59 to 0.73 for the DFS prediction model.ConclusionOur results, if confirmed, indicated that immune-related biomarkers should be taken into account or stratified in survival analysis for HCC.

Predicting the Risk of Human Immunodeficiency Virus Type 1 (HIV-1) Acquisition in Rural South Africa Using Geospatial Data.

BackgroundAccurate human immunodeficiency virus (HIV) risk assessment can guide optimal HIV prevention. We evaluated the performance of risk prediction models incorporating geospatial measures.MethodsWe developed and validated HIV risk prediction models in a population-based cohort in South Africa. Individual-level covariates included demographic and sexual behavior measures, and geospatial covariates included community HIV prevalence and viral load estimates. We trained models on 2012–2015 data using LASSO Cox models and validated predictions in 2016–2019 data. We compared full models to simpler models restricted to only individual-level covariates or only age and geospatial covariates. We compared the spatial distribution of predicted risk to that of high incidence areas (≥ 3/100 person-years).ResultsOur analysis included 19 556 individuals contributing 44 871 person-years and 1308 seroconversions. Incidence among the highest predicted risk quintile using the full model was 6.6/100 person-years (women) and 2.8/100 person-years (men). Models using only age group and geospatial covariates had similar performance (women: AUROC = 0.65, men: AUROC = 0.71) to the full models (women: AUROC = 0.68, men: AUROC = 0.72). Geospatial models more accurately identified high incidence regions than individual-level models; 20% of the study area with the highest predicted risk accounted for 60% of the high incidence areas when using geospatial models but only 13% using models with only individual-level covariates.ConclusionsGeospatial models with no individual measures other than age group predicted HIV risk nearly as well as models that included detailed behavioral data. Geospatial models may help guide HIV prevention efforts to individuals and geographic areas at highest risk.

Risk prediction models for discrete ordinal outcomes: Calibration and the impact of the proportional odds assumption.

Calibration is a vital aspect of the performance of risk prediction models, but research in the context of ordinal outcomes is scarce. This study compared calibration measures for risk models predicting a discrete ordinal outcome, and investigated the impact of the proportional odds assumption on calibration and overfitting. We studied the multinomial, cumulative, adjacent category, continuation ratio, and stereotype logit/logistic models. To assess calibration, we investigated calibration intercepts and slopes, calibration plots, and the estimated calibration index. Using large sample simulations, we studied the performance of models for risk estimation under various conditions, assuming that the true model has either a multinomial logistic form or a cumulative logit proportional odds form. Small sample simulations were used to compare the tendency for overfitting between models. As a case study, we developed models to diagnose the degree of coronary artery disease (five categories) in symptomatic patients. When the true model was multinomial logistic, proportional odds models often yielded poor risk estimates, with calibration slopes deviating considerably from unity even on large model development datasets. The stereotype logistic model improved the calibration slope, but still provided biased risk estimates for individual patients. When the true model had a cumulative logit proportional odds form, multinomial logistic regression provided biased risk estimates, although these biases were modest. Nonproportional odds models require more parameters to be estimated from the data, and hence suffered more from overfitting. Despite larger sample size requirements, we generally recommend multinomial logistic regression for risk prediction modeling of discrete ordinal outcomes.