Internal Validation Study Research Articles

From simple factors to artificial intelligence: evolution of prognosis prediction in childhood cancer: a systematic review and meta-analysis

Current paediatric cancer care requires innovative approaches to predict prognosis that facilitates personalised stratification, yet studies on the performance, composition and limitations of contemporary prognostic models are lacking. We aimed to compare the accuracy of traditional and advanced prognostic models. A systematic search for this systematic review and meta-analysis (CRTN42022370251) was conducted in PubMed, Embase, Scopus, and the Cochrane Library databases on 28 June 2024. Studies on the accuracy of prognostic markers or models used in paediatric haematological malignancies, central nervous system (CNS), or non-CNS solid tumours (NCNSST) were included. Three model categories were defined using: 1-clinical parameters, 2-genomic-transcriptomic data, and 3-artificial intelligence (AI). Primary outcomes were area under the receiver operating characteristic curve with a 95% confidence interval (CI) for various overall survival intervals and event-free survival. Two independent groups performed selection and data extraction. We used data published by the authors and publicly available databases. Of 12,982 studies, 358 were included in the meta-analysis and 27 in the systematic review, with limited data on AI-approaches. Most data were reported on NCNSST at 5-year OS, where a statistically significant difference was observed between Category-1 (0.75 CI: 0.72-0.79) and Category-2 (0.85 CI: 0.82-0.88) (p<0.001), but not between Categories-2 and -3 (p=0.2834) (0.82 CI: 0.77-0.88). Internal validation studies showed significantly better performance compared to those using external validation, highlighting the high risk of bias (ROB) inherent in internal validation. High ROB was most commonly experienced in the outcomes and statistical analysis domains, assessed using PROBAST and QUIPS. It is advisable to introduce Category-2 and -3 models in a clinical setting, especially for NCNSST prognostic for aiding risk-stratification. Although AI-supported predictions in paediatric oncology are at an early stage of development, it is imperative to further explore their potential. This requires structured data collection and ethical sharing from paediatric oncology patients in sufficient quantity and quality. None.

Simple severity scale for perforated peptic ulcer with generalized peritonitis: a derivation and internal validation study.

Perforated peptic ulcer (PPU) causes peritonitis and requires surgery based on disease severity. This study aimed to develop and validate a severity scale for PPU with generalized peritonitis. This retrospective cohort study used a nationwide multicenter surgical database (2013-2020). Patients aged >15 years who underwent surgery for PPU with generalized peritonitis were included and categorized into the derivation (2013-2018) and two validation (2019 and 2020) cohorts. Possible severity predictors were selected via a literature review, and Lasso models were developed to predict severe postoperative adverse events with 2000 bootstrapping. Final variables for the scoring system were determined based on inclusion frequency (≥90%) in the Lasso models. Discrimination and accuracy were evaluated using C-statistics and calibration plots. Cutoff values for minimal postoperative adverse events were examined using negative predictive values. Among 12513 patients included (1202 underwent laparoscopic surgery), 533 (5.9%), 138 (7.6%), and 117 (6.9%) in the derivation and two validation cohorts experienced postoperative adverse events. Age, dyspnea at rest, preoperative sepsis, III/IV/V of American Society of Anesthesiologists physical status, and albumin and creatinine were selected for the final model. A 0-11 scoring system was developed with C-statistics of 0.812-0.819. Cutoff value was determined as 5, which predicted <3% probability of postoperative adverse events regardless of type of surgery. A score of <5 predicts minimal risks for postoperative adverse events and, therefore, would be clinically useful to determine the type of surgery. Further studies are needed to validate the score.

Diagnostic performance of deep learning for infectious keratitis: a systematic review and meta-analysis

Infectious keratitis (IK) is the leading cause of corneal blindness globally. Deep learning (DL) is an emerging tool for medical diagnosis, though its value in IK is unclear. We aimed to assess the diagnostic accuracy of DL for IK and its comparative accuracy with ophthalmologists. In this systematic review and meta-analysis, we searched EMBASE, MEDLINE, and clinical registries for studies related to DL for IK published between 1974 and July 16, 2024. We performed meta-analyses using bivariate models to estimate summary sensitivities and specificities. This systematic review was registered with PROSPERO (CRD42022348596). Of 963 studies identified, 35 studies (136,401 corneal images from >56,011 patients) were included. Most studies had low risk of bias (68.6%) and low applicability concern (91.4%) in all domains of QUADAS-2, except the index test domain. Against the reference standard of expert consensus and/or microbiological results (seven external validation studies; 10,675 images), the summary estimates (95% CI) for sensitivity and specificity of DL for IK were 86.2% (71.6-93.9) and 96.3% (91.5-98.5). From 28 internal validation studies (16,059 images), summary estimates for sensitivity and specificity were 91.6% (86.8-94.8) and 90.7% (84.8-94.5). Based on seven studies (4007 images), DL and ophthalmologists had comparable summary sensitivity [89.2% (82.2-93.6) versus 82.2% (71.5-89.5); P=0.20] and specificity [(93.2% (85.5-97.0) versus 89.6% (78.8-95.2); P=0.45]. DL models may have good diagnostic accuracy for IK and comparable performance to ophthalmologists. These findings should be interpreted with caution due to the image-based analysis that did not account for potential correlation within individuals, relatively homogeneous population studies, lack of pre-specification of DL thresholds, and limited external validation. Future studies should improve their reporting, data diversity, external validation, transparency, and explainability to increase the reliability and generalisability of DL models for clinical deployment. NIH, Wellcome Trust, MRC, Fight for Sight, BHP, and ESCRS.

Public evidence on AI products for digital pathology

Novel products applying artificial intelligence (AI)-based methods to digital pathology images are touted to have many uses and benefits. However, publicly available information for products can be variable, with few sources of independent evidence. This review aimed to identify public evidence for AI-based products for digital pathology. Key features of products on the European Economic Area/Great Britain (EEA/GB) markets were examined, including their regulatory approval, intended use, and published validation studies. There were 26 AI-based products that met the inclusion criteria and, of these, 24 had received regulatory approval via the self-certification route as General in vitro diagnostic (IVD) medical devices. Only 10 of the products (38%) had peer-reviewed internal validation studies and 11 products (42%) had peer-reviewed external validation studies. To support transparency an online register was developed using identified public evidence (https://osf.io/gb84r/), which we anticipate will provide an accessible resource on novel devices and support decision making.

An Automated Malnutrition Screening Tool Using Routinely Collected Data for Older Adults in Long-Term Care: Development and Internal Validation of AutoMal

ObjectiveTo develop and internally validate a malnutrition screening tool based on routinely collected data in the long-term care setting. DesignDiagnostic prediction model development and internal validation study. Setting and ParticipantsResidents (n = 539) from 10 long-term care facilities in Australia. MethodsCandidate variables identified through expert consultation were collected from routinely collected data in a convenience sample of long-term care facilities. Logistic regression using the Subjective Global Assessment as the reference standard was conducted on 500 samples derived using bootstrapping from the original sample. Candidate variables were selected if included in more than 95% of samples using backwards stepwise elimination. The final model was developed using logistic regression of selected variables. Internal validation was conducted using bootstrapping to calculate the optimism-adjusted performance. Overall discrimination was evaluated via receiver operator characteristic curve and calculation of the area under the curve. Youden's Index was used to identify the optimal threshold value for classifying malnutrition. Sensitivity and specificity were calculated. ResultsBody mass index and weight change % over 6 months were included in the automated malnutrition screening model (AutoMal), identified in 100% of bootstrapped samples. AutoMal demonstrated excellent discrimination of malnutrition, with area under the curve of 0.8378 (95% CI, 0.80–0.87). Youden's Index value was 0.37, resulting in sensitivity of 78% (95% CI, 71%–83%) and specificity of 77% (72%–81%). Optimism-corrected area under the curve was 0.8354. Conclusions and ImplicationsThe AutoMal demonstrates excellent ability to differentiate malnutrition status. It makes automated identification of malnutrition possible by using 2 variables commonly found in electronic health records.

Maternal History of Childhood Maltreatment and Pregnancy Weight Outcomes.

Childhood maltreatment is associated with elevated adult weight. It is unclear whether this association extends to pregnancy, a critical window for the development of obesity. We examined associations of childhood maltreatment histories with prepregnancy body mass index (BMI) and gestational weight gain among women who had participated for >20 years in a longitudinal cohort. At age 26-35 years, participants reported childhood maltreatment (physical, sexual, and emotional abuse; emotional neglect) and, 5 years later, about prepregnancy weight and gestational weight gain for previous pregnancies (n = 656). Modified Poisson regression models were used to estimate associations of maltreatment history with prepregnancy BMI and gestational weight gain z -scores, adjusting for sociodemographics. We used multivariate imputation by chained equations to adjust outcome measures for misclassification using data from an internal validation study. Before misclassification adjustment, results indicated a higher risk of prepregnancy BMI ≥30 kg/m 2 in women with certain types of maltreatment (e.g., emotional abuse risk ratio = 2.4; 95% confidence interval: 1.5, 3.7) compared with women without that maltreatment type. After misclassification adjustment, estimates were attenuated but still modestly elevated (e.g., emotional abuse risk ratio = 1.7; 95% confidence interval: 1.1, 2.7). Misclassification-adjusted estimates for maltreatment associations with gestational weight gain z -scores were close to the null and imprecise. Findings suggest an association of maltreatment with prepregnancy BMI ≥30 kg/m 2 but not with high gestational weight gain. Results suggest a potential need for equitable interventions that can support all women, including those with maltreatment histories, as they enter pregnancy.

MSKI-RADS: An MRI-based Musculoskeletal Infection Reporting and Data System for the Diagnosis of Extremity Infections.

Background Current terms used to describe the MRI findings for musculoskeletal infections are nonspecific and inconsistent. Purpose To develop and validate an MRI-based musculoskeletal infection classification and scoring system. Materials and Methods In this retrospective cross-sectional internal validation study, a Musculoskeletal Infection Reporting and Data System (MSKI-RADS) was designed. Adult patients with radiographs and MRI scans of suspected extremity infections with a known reference standard obtained between June 2015 and May 2019 were included. The scoring categories were as follows: 0, incomplete imaging; I, negative for infection; II, superficial soft-tissue infection; III, deeper soft-tissue infection; IV, possible osteomyelitis (OM); V, highly suggestive of OM and/or septic arthritis; VI, known OM; and NOS (not otherwise specified), nonspecific bone lesions. Interreader agreement for 20 radiologists from 13 institutions (intraclass correlation coefficient [ICC]) and true-positive rates of MSKI-RADS were calculated and the accuracy of final diagnoses rendered by the readers was compared using generalized estimating equations for clustered data. Results Among paired radiographs and MRI scans from 208 patients (133 male, 75 female; mean age, 55 years ± 13 [SD]), 20 were category I; 34, II; 35, III; 30, IV; 35, V; 18, VI; and 36, NOS. Moderate interreader agreement was observed among the 20 readers (ICC, 0.70; 95% CI: 0.66, 0.75). There was no evidence of correlation between reader experience and overall accuracy (P = .94). The highest true-positive rate was for MSKI-RADS I and NOS at 88.7% (95% CI: 84.6, 91.7). The true-positive rate was 73% (95% CI: 63, 80) for MSKI-RADS V. Overall reader accuracy using MSKI-RADS across all patients was 65% ± 5, higher than final reader diagnoses at 55% ± 7 (P < .001). Conclusion MSKI-RADS is a valid system for standardized terminology and recommended management of imaging findings of peripheral extremity infections across various musculoskeletal-fellowship-trained reader experience levels. © RSNA, 2024 Supplemental material is available for this article. See also the editorial by Schweitzer in this issue.

Grading objective diagnostic components in paroxysmal events: One-year follow-up at a tertiary epilepsy center.

This study was undertaken to develop a model and perform a preliminary internal validation study of the Scale for Objective Diagnostic Components of Paroxysmal Events (STAMP). We developed STAMP, which builds on the International League Against Epilepsy task force scale for functional seizures with additional categories for epileptic seizures and syncope. We included 200 consecutive referrals to a Dutch tertiary epilepsy center to evaluate seizurelike events. We recorded demographic and clinical data and collected the clinical evaluation at referral and after 3, 6, 9, and 12 months of follow-up. We ascertained the STAMP at each time point and evaluated factors predicting an improvement in STAMP grade during follow-up. Of the 200 referrals at baseline, 131 were classified as having epileptic seizures, 17 as functional seizures, and three as syncope, and 49 were unclassifiable. STAMP grade at baseline was 4 (absent) in 56 individuals, 3 (circumstantial) in 78, 2 (clinically established) in six, and 1 (documented) in 11. Over time, 62 cases STAMP grades improved, and 23 remained unclassifiable. A refinement of STAMP grade during follow-up was due to successful event recordings in 34 people (30 video-electroencephalographic [EEG] recordings, four tilt table testing), home videos or clinician-witnessed events in 13, and identification of interictal EEG or magnetic resonance imaging abnormalities in seven. An improved STAMP grade after 12 months of follow-up was significantly more likely in those with higher event frequency, unclassifiable events, longer event duration, and a shorter time since the first event and less likely in those with a history suggestive of seizures. This epilepsy service evaluation underscores the crucial role of event recording in improving diagnostic certainty. STAMP may be used to monitor diagnostic performance over time but requires further validation.

Quiet Quitting reason and behavior scale design and development process.

The concept of "Quiet Quitting", which might be considered new in the literature, is used to describe a passive and silent giving up. The aim of this study is to develop a scale to evaluate both the causes and consequences of "quiet quitting" which is believed to be increasing for various reasons in our country as well as in the world. Five-stage scale development method which was presented by Cohen and Swerdik was used in the process of developing the Quiet Quitting Reason and Behavior Scale (QQRBS). First, the conceptual structure of the scale was determined. In the second stage, the scale was structured. Structuring the scale is deciding on the scale type and scaling technique. The third stage is the first application of the scale. After the scale was applied, factor analysis, internal consistency assessment and validity studies were carried out in the item analysis section, which is the fourth stage. The fifth stage is the second application stage of the scale and at this stage, the scale was reviewed by test-retest and an application guideline was prepared. Cronbach's alpha value of draft items is 0.87, quiet quitting behavior sub-dimension Cronbach's Alpha value is 0.76 and reasons for quiet quitting Cronbach's Alpha value is 0.820. It was decided to keep the remaining 35 items after EFA analysis because the corrected correlation numbers of the items were >0.30. This study explains the design and development process of the QQRBS in Turkish society. According to this developed scale, as the scale mean score of the total and sub-dimensions increases, the tendency to quiet quitting increases.

The Revised Reflux Symptom Index (R-RSI): development, internal and external validation study.

This study proposes a revised version of the Reflux Symptom Index (R-RSI), a seventeen-item questionnaire that was revised to increase the suspicion of laryngo-pharyngeal reflux disease (LPRD). Internal validation involved 213 participants, comprising160 subjects without a previous LPRD diagnosis and 53 subjects with a self-reported previous diagnosis of LPRD with or without gastro-esophageal reflux disease (GERD). Test-retest reliability and internal consistency were calculated. For the external validation, 56 patients (independent from the previous cohort) were enrolled to explore the R-RSI Screening Properties and determine a cut-off using 24-h MII-pH as the gold standard. R-RSI test-retest reliability was high, both for the total score (ICC: 0.970) and for each item (ranging from 0.876 to 0.980). Cronbach's alpha was 0.910, indicating excellent internal consistency of the questionnaire. Participants with a previous self-reported diagnosis scored significantly higher (mean 24.94 ± 7.4; median 26, IQR 20-29) than those without a previous diagnosis (mean 4.66 ± 5.3; median 4, IQR 1-6) (p-value &lt; 0.0001). Participants with both previous LPRD and GERD diagnoses had higher scores (27.20 ± 7.8) compared to those with only LPRD (21.77 ± 5.5) (p-value=0.003). Using 24-h MII-pH diagnosis as a gold standard, the optimal R-RSI cut-off point was determined to be 18, with a sensitivity of 84.5% and a specificity of 81.8%, positive predictive value of 95% and negative predictive value of 60%. Our results suggest that the R-RSI may be useful to suspect LPRD, with or without GERD. The R-RSI is a self-administered patient-reported outcome questionnaire that demonstrates excellent reliability and high screening properties. Employing a cut-off of ≥18 the R-RSI can assist in diagnosing and monitoring LPRD.

A clinical decision instrument for head CT in intoxicated patients presenting to the emergency department: the DITCH study

Performing head CT in intoxicated patients presenting to the emergency department (ED) is common, yet low yield. In our previous study, acute findings were found only in patients with at least one of: neurological Deficits, Intubation, Trauma, Convulsions or Headache. We hypothesise that the absence of these DITCH clinical features rules out the need for immediate CT. We collected data for intoxicated patients attending EDs in our district from March 2021 to May 2022 inclusive. We recorded clinical presentation features, agents, disposition and head CT results. There were 1,308 intoxicated patients out of a total of 219,273 ED presentations. Median age was 38 years. Common ingestions were ethanol, stimulants, GABA-ergics and opioids. 407 patients (31%) had head CT, with 31 patients (7.6%) having 36 acute findings: 19 intracranial haemorrhages, 8 fractures, 2 cerebral oedema and 7 haematomas. All patients with acute CT findings had one of the DITCH features, and all had trauma. The sensitivity and NPV of the hypothesis were 100%. In patients that were not scanned, there were no re-presentations.Our internal validation study is further evidence that acute findings on head CT in intoxicated patients presenting to the ED are only found in those who had one of the DITCH features.

Defining wound bed conformability: a new testing methodology to assess the relative swelling rise of foam dressings.

Using a dressing that expands and conforms to the wound bed upon exudate absorption is one of the best ways to promote wound healing. While many products claim wound bed conformability, no externally replicated or verified test methodology had been developed to quantify a wound dressing's ability to conform to the wound bed. The Relative Swelling Rise (RSR) test methodology was developed to measure the relative swelling rise of foam dressings upon fluid absorption, and offers a quantifiable and easily replicated method to measure wound bed conformability. The RSR test method was developed, validated and reliability tested by Coloplast A/S, Denmark. External replication was provided by ALS Odense, Denmark (previously DB Lab). Circular fences provide a fixed diameter to apply and contain the fluid and prevent horizontal spreading in the test set-up. The swelling height is quantified relative to the fence's inner diameter, i.e., the ratio alpha (α), and allows evaluation of a material's ability to conform to the wound bed. Biatain Silicone foam products (n=3, Coloplast A/S, Denmark) were tested, all afforded an average α-ratio from 0.30 to 0.60. The relative standard deviations were between 1-3%, demonstrating the strength of the test. Robustness of the methodology was demonstrated through the internal validation study, the reliability study, and both an internal and external replication study, as well as a systematic literature review and expert review of the construct, content, criterion and generalisability of the method. Having a validated, effective and easily replicable testing method to quantify wound bed conformability of foam dressings is an important step towards achieving better healing outcomes.

A developmental validation of the Quick TargSeq 1.0 integrated system for automated DNA genotyping in forensic science for reference samples.

Analysis of short tandem repeats (STRs) is a global standard method for human identification. Insertion/Deletion polymorphisms (DIPs) can be used for biogeographical ancestry inference. Current DNA typing involves a trained forensic worker operating several specialized instruments in a controlled laboratory environment, which takes 6-8h. We developed the Quick TargSeq 1.0 integrated system (hereinafter abbreviated to Quick TargSeq) for automated generation of STR and DIP profiles from buccal swab samples and blood stains. The system fully integrates the processes of DNA extraction, polymerase chain reaction (PCR) amplification, and electrophoresis separation using microfluidic biochip technology. Internal validation studies were performed using RTyper 21 or DIP 38 chip cartridges with single-source reference samples according to the Scientific Working Group for DNA Analysis Methods guidelines. These results indicated that the Quick TargSeq system can process reference samples and generate STR or DIP profiles in approximately 2h, and the profiles were concordant with those determined using traditional STR or DIP analysis methods. Thus, reproducible and concordant DNA profiles were obtained from reference samples. Throughout the study, no lane-to-lane or run-to-run contamination was observed. The Quick TargSeq system produced full profiles from buccal swabs with at least eight swipes, dried blood spot cards with two 2-mm disks, or 10ng of purified DNA. Potential PCR inhibitors (i.e., coffee, smoking tobacco, and chewing tobacco) did not appear to affect the amplification reactions of the instrument. The overall success rate and concordance rate of 153 samples were 94.12% and 93.44%, respectively, which is comparable to other commercially available rapid DNA instruments. A blind test initiated by a DNA expert group showed that the system can correctly produce DNA profiles with 97.29% genotype concordance with standard bench-processing methods, and the profiles can be uploaded into the national DNA database. These results demonstrated that the Quick TargSeq system can rapidly generate reliable DNA profiles in an automated manner and has the potential for use in the field and forensic laboratories.

Potential value of serum prealbumin and serum albumin in the identification of postoperative delirium in patients undergoing knee/hip replacement: an observational study and internal validation study.

Postoperative delirium (POD) is a common postoperative neurological complication that can lead to a variety of postoperative complications. At present, the pathogenesis of POD is unclear. This study aims to explore the relationship between serum prealbumin and serum albumin and POD and whether serum prealbumin and serum albumin influence POD through POD core pathology. We enrolled 500 Chinese Han patients between September 2020 to January 2023. We analyzed the risk and protective factors of POD using the multivariate logistic regression. We also assessed the predictive power of serum prealbumin, serum albumin, and both in combination with CSF POD biomarkers. We used Stata MP16.0. to examine whether the association between serum prealbumin and serum albumin and POD was mediated by CSF POD biomarkers, and conducted an internal validation study to verify the accuracy of the combination of serum prealbumin + serum albumin + CSF POD biomarkers for predicting POD. The model was visualized using ROC curve and decision curve analysis (DCA). DynNom and Shiny packages were used to create an online calculator. Ten patients who had POD occurring from February 2023 to October 2023 were selected for internal verification. Finally, a total of 364 patients were included in our study. Levels of serum prealbumin, serum albumin in the POD group were lower than those in the NPOD group. The lever of serum prealbumin, serum albumin were protective factors for POD. The relationship between serum prealbumin, serum albumin and POD was partially mediated by T-tau (12.28%) and P-tau (20.61%). The model combining serum prealbumin and serum albumin and POD biomarkers exhibited a relatively better discriminatory ability to predict POD. DCA also showed that the combination of serum prealbumin and serum albumin and POD biomarkers brought high predictive benefits to patients. The dynamic online calculator can accurately predict the occurrence of POD in the internal validation study. Preoperative low serum prealbumin and serum albumin levels were the preoperative risk factors for POD, which is partly mediated by T-tau and P-tau. The model combining serum prealbumin and serum albumin and CSF POD biomarkers can accurately predict the occurrence of POD. http://www.clinicaltrials.gov, identifier ChiCTR2000033439.

Performance of the streamlined quality outcomes database web-based calculator: internal and external validation

BACKGROUND CONTEXTWith an increasing number of web-based calculators designed to provide the probabilities of an individual achieving improvement after lumbar spine surgery, there is a need to determine the accuracy of these models. PURPOSETo perform an internal and external validation study of the reduced Quality Outcomes Database web-based Calculator (QOD-Calc). STUDY DESIGNObservational longitudinal cohort. PATIENT SAMPLEPatients enrolled study-wide in Quality Outcomes Database (QOD) and patients enrolled in DaneSpine at a single institution who had elective lumbar spine surgery with baseline data to complete QOD-Calc and 12-month postoperative data. OUTCOME MEASURESOswestry Disability Index (ODI), Numeric Rating Scales (NRS) for back and leg pain, EuroQOL-5D (EQ-5D). METHODSBaseline data elements were entered into QOD-Calc to determine the probability for each patient having Any Improvement and 30% Improvement in NRS leg pain, back pain, EQ-5D and ODI. These probabilities were compared with the actual 12-month postop data for each of the QOD and DaneSpine cases. Receiver-operating characteristics analyses were performed and calibration plots created to assess model performance. RESULTS24,755 QOD cases and 8,105 DaneSpine lumbar cases were included in the analysis. QOD-Calc had acceptable to outstanding ability (AUC: 0.694–0.874) to predict Any Improvement in the QOD cohort and moderate to acceptable ability (AUC: 0.658–0.747) to predict 30% Improvement. QOD-Calc had acceptable to exceptional ability (AUC: 0.669–0.734) to predict Any improvement and moderate to exceptional ability (AUC: 0.619–0.862) to predict 30% Improvement in the DaneSpine cohort. AUCs for the DaneSpine cohort was consistently lower that the AUCs for the QOD validation cohort. CONCLUSIONQOD-Calc performs well in predicting outcomes in a patient population that is similar to the patients that was used to develop it. Although still acceptable, model performance was slightly worse in a distinct population, despite the fact that the sample was more homogenous. Model performance may also be attributed to the low discrimination threshold, with close to 90% of cases reporting Any Improvement in outcome. Prediction models may need to be developed that are highly specific to the characteristics of the population.

Predicting anticipated benefit from an extended consultation to personalise care in multimorbidity: a development and internal validation study of a prioritisation algorithm in general practice.

Persons with multimorbidity may gain from person-centred care compared with the current protocolised chronic-disease management in Dutch general practice. Given time constraints and limited resources, it is essential to prioritise those most in need of an assessment of person-centred chronic-care needs. To develop and validate a prioritisation algorithm based on routine electronic medical record (EMR) data that distinguishes between patients with multimorbidity who would, and those who would not, benefit from an extended person-centred consultation to assess person-centred chronic-care needs, as judged by GPs. A mixed-methods study was conducted in five general practices in the north-west region of the Netherlands. Four out of the five practices were situated in rural areas. Multivariable logistic regression using EMR data to predict the GPs' judgement on patients' anticipated benefit from an extended consultation, as well as a thematic analysis of a focus group exploring GPs' clinical reasoning for this judgement were conducted. Internal validation was performed using 10-fold cross-validation. Multimorbidity was defined as the presence of ≥3 chronic conditions. In total, EMRs from 1032 patients were included in the analysis; of these, 352 (34.1%) were judged to have anticipated benefit. The model's cross-validated C-statistic was 0.72 (95% confidence interval = 0.70 to 0.75). Calibration was good. Presence of home visit(s) and history of myocardial infarction were associated with anticipated benefit. Thematic analysis revealed three dimensions feeding anticipated benefit: GPs' cause for concern, patients' mindset regarding their conditions, and balance between received care/expected care needed. This algorithm may facilitate automated prioritisation, potentially avoiding the need for GPs to personally triage the whole practice population that has multimorbidity. However, external validation of the algorithm and evaluation of actual benefit of consultation is recommended before implementation.

Predicting 1-, 3-, 5-, and 8-year all-cause mortality in a community-dwelling older adult cohort: relevance for predictive, preventive, and personalized medicine

BackgroundPopulation aging is a global public health issue involving increased prevalence of age-related diseases, and concomitant burden on medical resources and the economy. Ninety-two diseases have been identified as age-related, accounting for 51.3% of the global adult disease burden. The economic cost per capita for older people over 60 years is 10 times that of the younger population. From the aspects of predictive, preventive, and personalized medicine (PPPM), developing a risk-prediction model can help identify individuals at high risk for all-cause mortality and provide an opportunity for targeted prevention through personalized intervention at an early stage. However, there is still a lack of predictive models to help community-dwelling older adults do well in healthcare.ObjectivesThis study aims to develop an accurate 1-, 3-, 5-, and 8-year all-cause mortality risk-prediction model by using clinical multidimensional variables, and investigate risk factors for 1-, 3-, 5-, and 8-year all-cause mortality in community-dwelling older adults to guide primary prevention.MethodsThis is a two-center cohort study. Inclusion criteria: (1) community-dwelling adult, (2) resided in the districts of Chaonan or Haojiang for more than 6 months in the past 12 months, and (3) completed a health examination. Exclusion criteria: (1) age less than 60 years, (2) more than 30 incomplete variables, (3) no signed informed consent. The primary outcome of the study was all-cause mortality obtained from face-to-face interviews, telephone interviews, and the medical death database from 2012 to 2021. Finally, we enrolled 5085 community-dwelling adults, 60 years and older, who underwent routine health screening in the Chaonan and Haojiang districts, southern China, from 2012 to 2021. Of them, 3091 participants from Chaonan were recruited as the primary training and internal validation study cohort, while 1994 participants from Haojiang were recruited as the external validation cohort. A total of 95 clinical multidimensional variables, including demographics, lifestyle behaviors, symptoms, medical history, family history, physical examination, laboratory tests, and electrocardiogram (ECG) data were collected to identify candidate risk factors and characteristics. Risk factors were identified using least absolute shrinkage and selection operator (LASSO) models and multivariable Cox proportional hazards regression analysis. A nomogram predictive model for 1-, 3-, 5- and 8-year all-cause mortality was constructed. The accuracy and calibration of the nomogram prediction model were assessed using the concordance index (C-index), integrated Brier score (IBS), receiver operating characteristic (ROC), and calibration curves. The clinical validity of the model was assessed using decision curve analysis (DCA).ResultsNine independent risk factors for 1-, 3-, 5-, and 8-year all-cause mortality were identified, including increased age, male, alcohol status, higher daily liquor consumption, history of cancer, elevated fasting glucose, lower hemoglobin, higher heart rate, and the occurrence of heart block. The acquisition of risk factor criteria is low cost, easily obtained, convenient for clinical application, and provides new insights and targets for the development of personalized prevention and interventions for high-risk individuals. The areas under the curve (AUC) of the nomogram model were 0.767, 0.776, and 0.806, and the C-indexes were 0.765, 0.775, and 0.797, in the training, internal validation, and external validation sets, respectively. The IBS was less than 0.25, which indicates good calibration. Calibration and decision curves showed that the predicted probabilities were in good agreement with the actual probabilities and had good clinical predictive value for PPPM.ConclusionThe personalized risk prediction model can identify individuals at high risk of all-cause mortality, help offer primary care to prevent all-cause mortality, and provide personalized medical treatment for these high-risk individuals from the PPPM perspective. Strict control of daily liquor consumption, lowering fasting glucose, raising hemoglobin, controlling heart rate, and treatment of heart block could be beneficial for improving survival in elderly populations.

OC 36.2 Machine-Learning Based Risk Prediction of Cancer Associated Thrombosis: Internal, External, and Pre-Clinical Validation Studies

OC 36.2 Machine-Learning Based Risk Prediction of Cancer Associated Thrombosis: Internal, External, and Pre-Clinical Validation Studies

Transportability of bacterial infection prediction models for critically ill patients.

Bacterial infections (BIs) are common, costly, and potentially life-threatening in critically ill patients. Patients with suspected BIs may require empiric multidrug antibiotic regimens and therefore potentially be exposed to prolonged and unnecessary antibiotics. We previously developed a BI risk model to augment practices and help shorten the duration of unnecessary antibiotics to improve patient outcomes. Here, we have performed a transportability assessment of this BI risk model in 2 tertiary intensive care unit (ICU) settings and a community ICU setting. We additionally explored how simple multisite learning techniques impacted model transportability. Patients suspected of having a community-acquired BI were identified in 3 datasets: Medical Information Mart for Intensive Care III (MIMIC), Northwestern Medicine Tertiary (NM-T) ICUs, and NM "community-based" ICUs. ICU encounters from MIMIC and NM-T datasets were split into 70/30 train and test sets. Models developed on training data were evaluated against the NM-T and MIMIC test sets, as well as NM community validation data. During internal validations, models achieved AUROCs of 0.78 (MIMIC) and 0.81 (NM-T) and were well calibrated. In the external community ICU validation, the NM-T model had robust transportability (AUROC 0.81) while the MIMIC model transported less favorably (AUROC 0.74), likely due to case-mix differences. Multisite learning provided no significant discrimination benefit in internal validation studies but offered more stability during transport across all evaluation datasets. These results suggest that our BI risk models maintain predictive utility when transported to external cohorts. Our findings highlight the importance of performing external model validation on myriad clinically relevant populations prior to implementation.

Machine-Learning-Based Tool to Predict Target Prostate Biopsy Outcomes: An Internal Validation Study

The aim of this study is to present a personalized predictive model (PPM) with a machine learning (ML) system that is able to identify and classify patients with suspected prostate cancer (PCa) following mpMRI. We extracted all the patients who underwent fusion biopsy (FB) from March 2014 to December 2019, while patients from August 2020 to April 2021 were included as a validation set. The proposed system was based on the following four ML methods: a fuzzy inference system (FIS), the support vector machine (SVM), k-nearest neighbors (KNN), and self-organizing maps (SOMs). Then, a system based on fuzzy logic (FL) + SVM was compared with logistic regression (LR) and standard diagnostic tools. A total of 1448 patients were included in the training set, while 181 patients were included in the validation set. The area under the curve (AUC) of the proposed FIS + SVM model was comparable with the LR model but outperformed the other diagnostic tools. The FIS + SVM model demonstrated the best performance, in terms of negative predictive value (NPV), on the training set (78.5%); moreover, it outperformed the LR in terms of specificity (92.1% vs. 83%). Considering the validation set, our model outperformed the other methods in terms of NPV (60.7%), sensitivity (90.8%), and accuracy (69.1%). In conclusion, we successfully developed and validated a PPM tool using the FIS + SVM model to calculate the probability of PCa prior to a prostate FB, avoiding useless ones in 15% of the cases.