Model Validation Studies Research Articles

Predicting temporomandibular disorders in adults using interpretable machine learning methods: a model development and validation study

IntroductionTemporomandibular disorders (TMD) have a high prevalence and complex etiology. The purpose of this study was to apply a machine learning (ML) approach to identify risk factors for the occurrence of TMD in adults and to develop and validate an interpretable predictive model for the risk of TMD in adults.MethodsA total of 949 adults who underwent oral examinations were enrolled in our study. 5 different ML algorithms were used for model development and comparison, and feature selection was performed by feature importance ranking and feature decreasing methods. Several evaluation indexes, including the area under the receiver-operating-characteristic curve (AUC), were used to compare the predictive performance. The precision-recall curve (PR), calibration curve, and decision curve analysis (DCA) further assessed the accuracy and clinical utility of the model.ResultsThe performance of the random forest (RF) model was the best among the 5 ML models. An interpretable RF model was developed with 7 features (gender, malocclusion, unilateral chewing, chewing hard substances, grinding teeth, clenching teeth, and anxiety). The AUCs of the final model on the training set, internal validation set, and external test set were 0.892, 0.854, and 0.857, respectively. Calibration and DCA curves showed high accuracy and clinical applicability of the model.DiscussionAn efficient and interpretable TMD risk prediction model for adults was successfully developed using the ML method. The model not only has good predictive performance, but also enhances the clinical application value of the model through the SHAP method. This model can provide clinicians with a practical and efficient TMD risk assessment tool that can help them better predict and assess TMD risk in adults, supporting more efficient disease management and targeted medical interventions.

Artificial intelligence-enabled electrocardiogram for mortality and cardiovascular risk estimation: a model development and validation study

Artificial intelligence (AI)-enabled electrocardiography (ECG) can be used to predict risk of future disease and mortality but has not yet been adopted into clinical practice. Existing model predictions do not have actionability at an individual patient level, explainability, or biological plausibi. We sought to address these limitations of previous AI-ECG approaches by developing the AI-ECG risk estimator (AIRE) platform. The AIRE platform was developed in a secondary care dataset (Beth Israel Deaconess Medical Center [BIDMC]) of 1 163 401 ECGs from 189 539 patients with deep learning and a discrete-time survival model to create a patient-specific survival curve with a single ECG. Therefore, AIRE predicts not only risk of mortality, but also time-to-mortality. AIRE was validated in five diverse, transnational cohorts from the USA, Brazil, and the UK (UK Biobank [UKB]), including volunteers, primary care patients, and secondary care patients. AIRE accurately predicts risk of all-cause mortality (BIDMC C-index 0·775, 95% CI 0·773-0·776; C-indices on external validation datasets 0·638-0·773), future ventricular arrhythmia (BIDMC C-index 0·760, 95% CI 0·756-0·763; UKB C-index 0·719, 95% CI 0·635-0·803), future atherosclerotic cardiovascular disease (0·696, 0·694-0·698; 0·643, 0·624-0·662), and future heart failure (0·787, 0·785-0·789; 0·768, 0·733-0·802). Through phenome-wide and genome-wide association studies, we identified candidate biological pathways for the prediction of increased risk, including changes in cardiac structure and function, and genes associated with cardiac structure, biological ageing, and metabolic syndrome. AIRE is an actionable, explainable, and biologically plausible AI-ECG risk estimation platform that has the potential for use worldwide across a wide range of clinical contexts for short-term and long-term risk estimation. British Heart Foundation, National Institute for Health and Care Research, and Medical Research Council.

Machine Learning-Based Prediction of Neurodegenerative Disease in Patients With Type 2 Diabetes by Derivation and Validation in 2 Independent Korean Cohorts: Model Development and Validation Study.

Several machine learning (ML) prediction models for neurodegenerative diseases (NDs) in type 2 diabetes mellitus (T2DM) have recently been developed. However, the predictive power of these models is limited by the lack of multiple risk factors. This study aimed to assess the validity and use of an ML model for predicting the 3-year incidence of ND in patients with T2DM. We used data from 2 independent cohorts-the discovery cohort (1 hospital; n=22,311) and the validation cohort (2 hospitals; n=2915)-to predict ND. The outcome of interest was the presence or absence of ND at 3 years. We selected different ML-based models with hyperparameter tuning in the discovery cohort and conducted an area under the receiver operating characteristic curve (AUROC) analysis in the validation cohort. The study dataset included 22,311 (discovery) and 2915 (validation) patients with T2DM recruited between 2008 and 2022. ND was observed in 133 (0.6%) and 15 patients (0.5%) in the discovery and validation cohorts, respectively. The AdaBoost model had a mean AUROC of 0.82 (95% CI 0.79-0.85) in the discovery dataset. When this result was applied to the validation dataset, the AdaBoost model exhibited the best performance among the models, with an AUROC of 0.83 (accuracy of 78.6%, sensitivity of 78.6%, specificity of 78.6%, and balanced accuracy of 78.6%). The most influential factors in the AdaBoost model were age and cardiovascular disease. This study shows the use and feasibility of ML for assessing the incidence of ND in patients with T2DM and suggests its potential for use in screening patients. Further international studies are required to validate these findings.

Development of Machine Learning-Based Biopsy Algorithm for Kidney: Multicenter-Based Model Development and Validation Study in South Korea

Development of Machine Learning-Based Biopsy Algorithm for Kidney: Multicenter-Based Model Development and Validation Study in South Korea

Risk Prediction Models for Adverse Drug Reactions and Adverse Drug Events in Older Adults – A Systematic Review and Meta-analysis

Abstract Background Adverse drug reactions (ADRs) and adverse drug events (ADEs) are common and result in significant morbidity, mortality and healthcare costs. As the world’s population ages, the consumption of medications and healthcare will increase. Models predicting ADR risks in older adults have previously been found to lack reliability and validity. The aim of this systematic review and meta-analysis is to provide an updated, comprehensive quality assessment and analysis of ADR-risk prediction tools in older adults. Methods Standard computerised-databases and citations were searched (2012 to 2023) to identify relevant peer-reviewed studies. Studies which developed and/or validated an ADR/ADE prediction model for use in older adults were included. Four studies from a previous systematic review were also included. The TRIPOD (transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) guidelines were used to evaluate each of the included studies. Random effects models were used to derive a pooled discrimination estimate (area under the receiver operation curve; AUROC) for model development studies and model validation studies. The prediction model risk of bias assessment tool (PROBAST) was applied to evaluate bias. Results Five out of 11,481 titles, plus four previous studies, met all inclusion criteria and underwent TRIPOD evaluation. Meta-analysis resulted in a favourable pooled AUROC 0.75 (95% CI 0.57, 0.87; I-squared= 96.65%) for model development studies and 0.70 (95% CI 0.57, 0.80; I-squared= 90.53%) for model validation studies but there was substantial heterogeneity. Studies had poor adherence (range 32-50%; median 44%; IQR 11.5%) to TRIPOD guidelines. The overall risk of bias was high for all included studies. Conclusion Seven risk prediction models were identified, all exhibiting poor adherence to TRIPOD guidelines which may question the investigational rigor of the studies and their usability. This underscores the need for the development of a validated, robust and reliable tool worthy of implementation and testing in a real-world setting.

Microvascular effects of a mixed meal tolerance test: a model validation study

AbstractPurposeEndothelial dysfunction is a pathophysiological change preceding many cardiovascular events. Measuring improvements of endothelial function is challenging when function is already optimal, which may be remediated using a physiological challenge. This study aimed to determine whether imaging assessments can detect microvascular effects of a mixed meal tolerance test (MMTT).MethodsTwenty healthy volunteers (age ≥45 and ≤70 years) underwent two MMTTs at the beginning (Day 1) and end (Day 84) of a twelve‐week period. Imaging methods included laser speckle contrast imaging (LSCI) combined with post‐occlusive reactive hyperaemia (PORH) and local thermal hyperaemia (LTH) challenges, passive leg movement ultrasonography (PLM), and sidestream dark field microscopy (SDFM). Measurements were conducted pre‐MMTT and at 5 timepoints post‐MMTT for PLM and SDFM and 3 timepoints post‐MMTT for PORH and LTH.ResultsNo consistent effects of the MMTT were detected on LSCI LTH, PLM and SDFM endpoints. LSCI PORH maximum perfusion was significantly suppressed 46, 136, and 300 min post‐MMTT administration on Day 1, while residual perfusion decreased significantly 46 and 136 min post‐MMTT on Day 1. However, when repeated on Day 84, PORH endpoints were not significantly affected by the MMTT.ConclusionSDFM, PLM and LSCI LTH endpoints displayed high intra‐subject variability and did not detect consistent effects of MMTT. LSCI PORH endpoints displayed the lowest intra‐subject variability of all assessed endpoints and were affected by the MMTT on Day 1, but not on Day 84. Further standardization of methods or more robust challenges to affect vascular endpoints may be needed.

Musculoskeletal models determine the effect of a soft active exosuit on muscle activations and forces during lifting and lowering tasks

Exosuits have the potential to mitigate musculoskeletal stress and prevent back injuries during industrial tasks. This study aimed to 1) validate the implementation of a soft active exosuit into a musculoskeletal model of the spine by comparing model predicted muscle activations versus corresponding surface EMG measurements, and 2) evaluate the effect of the exosuit on peak back and hip muscle forces. Fourteen healthy participants performed squat and stoop lift and lower tasks with boxes of 6 and 10 kg, with and without wearing a 2.7 kg soft active exosuit. Participant-specific musculoskeletal models, which included the exosuit, were created in OpenSim. Model validation focused on the back and hip extensors, where temporal agreement between EMG and model estimated muscle activity was generally strong to excellent (average cross-correlation coefficients ranging from 0.84 to 0.98). Root mean square errors of muscle activity (0.05–0.10) were similar with and without the exosuit, and compared well to prior model validation studies without the exosuit (average root mean square errors ranging from 0.05 to 0.19). In terms of performance, the exosuit reduced the estimated peak erector spinae forces during lifting and lowering phases across all lifting tasks but reduced peak hip extensor muscles forces only in a squat lift task of 10 kg. These reductions in total peak muscle forces were approximately 1.7–4.2 times greater than the corresponding exosuit assistance force, which were 146 ± 19 N and 102 ± 14 N at the times of peak erector spinae forces in lifting and lowering, respectively. Overall, the results support the hypothesis that exosuits reduce soft tissue loading, and thereby potentially reduce fatigue and injury risk during manual materials handling tasks. Incorporating exosuits into musculoskeletal models is a valid approach to understand the impact of exosuit assistance on muscle activity and forces.

Validation of the HAS-BLED scale for the assessment of bleeding risk in patients on anticoagulation therapy with a diagnosis of venous thromboembolic disease.

Several models have been developed to assess bleeding risk in patients with venous thromboembolism, such as HAS-BLED, but their external validity has not been adequately assessed. The objective of the study was to evaluate the discriminative ability and calibration of the HAS-BLED scale for predicting 1-month bleeding risk in patient's anticoagulated for venous thromboembolism. External validation study of a prediction model based on a retrospective cohort of patients with venous thromboembolism treated between November 2019 and January 2022. Calibration of the HAS-BLED scale was evaluated using the Hosmer-Lemeshow test and the ratio of observed to expect events within each risk category. Discriminatory ability was assessed using the area under the curve (AUC) of a receiver operating characteristic curve. We included 735 patients (median age 64 years, female sex 55.2%), pulmonary embolism was diagnosed in most patients (60.7%), and 4.9% presented bleeding events. Regarding calibration, the HAS-BLED scale systematically underestimates the risk both in the general population (ROE 3.76, p < 0.001) and in cancer patients (ROE 4.16). The Hosmer-Lemeshow test rejected the hypothesis of adequate calibration (p < 0.001). Discriminatory ability was limited both in the general population (AUC = 0.57, 95% confidence interval [CI]: 0.48-0.66) and in the subgroup with active cancer (AUC = 0.53, 95% CI: 0.36-0.69). The HAS-BLED scale in patients with venous thromboembolism underestimates the risk of bleeding at 1 month and has a low ability to discriminate high-risk patients. Cautious interpretation of the scale is recommended until additional evidence is available.

Enhanced machine learning approaches for OSA patient screening: model development and validation study

Age, gender, body mass index (BMI), and mean heart rate during sleep were found to be risk factors for obstructive sleep apnea (OSA), and a variety of methods have been applied to predict the occurrence of OSA. This study aimed to develop and evaluate OSA prediction models using simple and accessible parameters, combined with multiple machine learning algorithms, and integrate them into a cloud-based mobile sleep medicine management platform for clinical use. The study data were obtained from the clinical records of 610 patients who underwent polysomnography (PSG) at the Sleep Medicine Center of the Second Affiliated Hospital of Fujian Medical University between January 2021 and December 2022. The participants were randomly divided into a training-test group (80%) and an independent validation group (20%). The logistic regression, artificial neural network, naïve Bayes, support vector machine, random forest, and decision tree algorithms were used with age, gender, BMI, and mean heart rate during sleep as predictors to build a risk prediction model for moderate-to-severe OSA. To evaluate the performance of the models, we calculated the area under the receiver operating curve (AUROC), accuracy, recall, specificity, precision, and F1-score for the independent validation set. In addition, the calibration curve, decision curve, and clinical impact curve were generated to determine clinical usefulness. Age, gender, BMI, and mean heart rate during sleep were significantly associated with OSA. The artificial neural network model had the best efficacy compared with the other prediction algorithms. The AUROC, accuracy, recall, specificity, precision, F1-score, and Brier score were 80.4% (95% CI 76.7–84.1%), 69.9% (95% CI 69.8–69.9%), 86.5% (95% CI 81.6–91.3%), 61.5% (95% CI 56.6–66.4%), 53.2% (95% CI 47.7–58.7%), 65.9% (95% CI 60.2–71.5%), and 0.165, respectively, for the artificial neural network model. The AUROCs for the LR, NB, SVM, RF, and DT models were 80.2%, 79.7%, 79.2%, 78.4%, and 70.4%, respectively. The six models based on four simple and easily accessible parameters effectively predicted moderate-to-severe OSA in patients with PSG screening, with the artificial neural network model having the best performance. These models can provide a reliable tool for early OSA diagnosis, and their integration into a cloud-based mobile sleep medicine management platform could improve clinical decision making.

Conditional generative adversarial network-assisted system for radiation-free evaluation of scoliosis using a single smartphone photograph: a model development and validation study

Adolescent idiopathic scoliosis (AIS) is the most common spinal disorder in children, characterized by insidious onset and rapid progression, which can lead to severe consequences if not detected in a timely manner. Currently, the diagnosis of AIS primarily relies on X-ray imaging. However, due to limitations in healthcare access and concerns over radiation exposure, this diagnostic method cannot be widely adopted. Therefore, we have developed and validated a screening system using deep learning technology, capable of generating virtual X-ray images (VXI) from two-dimensional Red Green Blue (2D-RGB) images captured by a smartphone or camera to assist spine surgeons in the rapid, accurate, and non-invasive assessment of AIS. We included 2397 patients with AIS and 48 potential patients with AIS who visited four medical institutions in mainland China from June 11th 2014 to November 28th 2023. Participants data included standing full-spine X-ray images captured by radiology technicians and 2D-RGB images taken by spine surgeons using a camera. We developed a deep learning model based on conditional generative adversarial networks (cGAN) called Swin-pix2pix to generate VXI on retrospective training (n=1842) and validation (n=100) dataset, then validated the performance of VXI in quantifying the curve type and severity of AIS on retrospective internal (n=100), external (n=135), and prospective test datasets (n=268). The prospective test dataset included 268 participants treated in Nanjing, China, from April 19th, 2023, to November 28th, 2023, comprising 220 patients with AIS and 48 potential patients with AIS. Their data underwent strict quality control to ensure optimal data quality and consistency. Our Swin-pix2pix model generated realistic VXI, with the mean absolute error (MAE) for predicting the main and secondary Cobb angles of AIS significantly lower than other baseline cGAN models, at 3.2° and 3.1° on prospective test dataset. The diagnostic accuracy for scoliosis severity grading exceeded that of two spine surgery experts, with accuracy of 0.93 (95% CI [0.91, 0.95]) in main curve and 0.89 (95% CI [0.87, 0.91]) in secondary curve. For main curve position and curve classification, the predictive accuracy of the Swin-pix2pix model also surpassed that of the baseline cGAN models, with accuracy of 0.93 (95% CI [0.90, 0.95]) for thoracic curve and 0.97 (95% CI [0.96, 0.98]), achieving satisfactory results on three external datasets as well. Our developed Swin-pix2pix model holds promise for using a single photo taken with a smartphone or camera to rapidly assess AIS curve type and severity without radiation, enabling large-scale screening. However, limited data quality and quantity, a homogeneous participant population, and rotational errors during imaging may affect the applicability and accuracy of the system, requiring further improvement in the future. National Key R&D Program of China, Natural Science Foundation of Jiangsu Province, China Postdoctoral Science Foundation, Nanjing Medical Science and Technology Development Foundation, Jiangsu Provincial Key Research and Development Program, and Jiangsu Provincial Medical Innovation Centre of Orthopedic Surgery.

Optimizing heat transfer predictions in HCNG engines: A novel model validation and comparative study via quasi-dimensional combustion modeling and artificial neural networks

Heat transfer from the walls of engine has a significant role on engine combustion, performance and emission characteristics. The study objectives to showcase the efficacy of the new model through an analysis by comparison with existing heat transfer models. New model is based on woschni model in which pressure and temperature is replaced by other fluid properties like density, thermal conductivity and dynamic viscosity. A series of experiments were conducted on a compressed natural gas internal combustion engine across varying hydrogen fractions, EGR ratios, engine speeds and different loads under stoichiometric conditions. The study demonstrates the efficacy of the proposed model by constantly achieving high prediction quality across a wide range of engine calibration coefficients by using Quasi-dimensional Combustion Model (QDCM) on MATLAB. Comparative analyses of new model with different heat transfer models were undertaken to validate the heat transfer rates with experimental results across a broad spectrum of operational conditions. It is observed that heat transfer rate is increased by increasing the engine load as 25%, 50%, 75% and 100% as 90.57J/deg, 130.12J/deg, 200.02J/deg, and 260.26J/deg with new model correspondingly. Heat transfer rate reduced by rise in engine speed with 1100 rpm, 1200 rpm, 1500 rpm and 1700 rpm is as 32.91 kW, 32.16 kW, 25.36 kW and 18.03 kW by new heat transfer model respectively. Artificial neural network (ANN) popular backpropagation algorithm is adopted to predict the heat transfer rate of HCNG engine, the five-input and one-output network structure are used. The values of correlation coefficient (R) and mean square error (MSE) were 0.99957 and 0.22667, 0.99998 and 0.010776, 0.99253 and 4.4762, 0.9961 and 1.2329, 0.99994 and 0.025108 for Woschni, New_Model, Chang, Sitkei and experimental respectively. This research work offers that ANN is a wise option for conventional modeling systems. In this way, the heat transfer rate of hydrogen-added CNG engines may be precisely predicted using ANN modeling.

Experimental validation study of equivalent bare charge calculation model for two shelled warheads

After the explosion of the warhead shell fragments will consume a portion of the charge release energy, a method of calculating the shockwave overpressure value of the explosion of the warhead is to calculate the shockwave overpressure value by firstly converting the actual charge C of the warhead into the charge CEB of the bare charge, and then converting CEB into TNT equivalent. Experimental results were used to compare the calculation error size of the two models for calculating the equivalent bare charge of the warhead, and the results show that the calculation results of Chi Jiachun's modified model are in better conformity with the experimental results; by analyzing the influence of the empirical constant in Chi Jiachun's modified model on the calculation results, the value of the empirical constant is given as a suggestion.

Validation and parametric study of FFRD model in DRACCAR code

The recent revision of Emergency Core Cooling System (ECCS) regulations in Korea has necessitated the consideration of Fuel Fragmentation, Relocation, and Dispersal (FFRD) phenomena in nuclear reactor safety analyses. Consequently, the Korea Atomic Energy Research Institute (KAERI) has developed and integrated the FFRD model into the domestically licensed safety analysis code, SPACE. Globally, US NRC’s FRAPTRAN and IRSN’s DRACCAR are available for evaluating FFRD phenomena. The FRAPTRAN code incorporates the FFR model, developed by Quantum Technology, while the fuel dispersal model is currently not included. In contrast, the DRACCAR code functions as an integrated analysis platform capable of modeling multi-dimensional thermal, hydraulic, mechanical, and chemical phenomena during a Loss of Coolant Accident (LOCA). This study conducts a thorough examination of the FFRD model in the DRACCAR code and validates its applicability through analysis using the Halden IFA-650 tests. The results demonstrate satisfactory predictive capabilities. Furthermore, a parametric study of key FFRD model parameters enhances the understanding of the FFRD model in the DRACCAR code. The development of detailed physical models in the FFRD model could significantly enhance the performance of the DRACCAR code, warranting the establishment of a comprehensive framework for these advancements. In the future, code-to-code comparisons between the DRACCAR code and other domestically developed integrated analysis platforms will be conducted to investigate various phenomena in depth.

Domain-Linked and Domain-Specific Competence: a Validation Study of a Two-Dimensional Model of Economic Vocational Competence in Germany

Modeling vocational competence is increasingly crucial for monitoring and enhancing the quality of Vocational Educational Training (VET), particularly in the context of ongoing international comparative studies known as "large-scale assessments" of vocational education and training. This study endeavors to provide well-structured and guideline-compliant empirical evidence for the validation of the two-dimensional construct of economic vocational competence, advancing beyond the current state of research. A sample of 1438 first-year apprentices from two federal states in Germany participated as test-takers. The authentic assessment framework comprised 24 items, assessing two dimensions of vocational competence: domain-linked competence and domain-specific competence in the business/commercial domain. Measurement invariance was assessed across (1) federal states and (2) versions of test booklets, and the Multidimensional Random Coefficient Multinomial Logit model was employed to examine the quality of the two-dimensional vocational competence construct. The results supported the validity of the structure, highlighting the differentiation between domain-linked competence and domain-specific competence. This provides a more substantively accurate representation of trainees' vocational competence compared to a unidimensional model.

Solar Wind Interactions with Comet C/2021 A1 Using STEREO HI and a Data-assimilative Solar Wind Model

Cometary tails display dynamic behavior attributed to interactions with solar wind structures. Consequently, comet-tail observations can serve as in situ solar wind monitors. During 2021 December, Comet Leonard (C/2021 A1) was observed by the STEREO-A heliospheric imager. The comet tail exhibited various signatures of interactions with the solar wind including bending, kink formation, and finally complete disconnection. In this study, we compare the timing of these events with solar wind structures predicted by the Heliospheric Upwind eXtrapolation model with a time-dependency (or HUXt) solar wind model using new solar wind data assimilation (DA) techniques. This serves both to provide the most accurate solar wind context to interpret the cometary processes, but also as a test of the DA and an example of how comet observations can be used in model validation studies. Coronal mass ejections, stream interaction regions (SIRs), and heliospheric current sheet (HCS) crossings were all considered as potential causes of the tail disconnection. The results suggest the tail disconnection at Comet Leonard was the result of it crossing the HCS and into an SIR. The timing of these features agree better with the DA model results than the non-DA model, showing the value of this approach. Case studies such as this expand our understanding of comet–solar wind interactions, and in demonstrating the utility of DA for solar wind modeling. We note that this could lead to comets acting as additional in situ measures for solar wind conditions for regions where no in situ spacecraft are available, potentially improving solar wind DA in the future.

Detection of Common Respiratory Infections, Including COVID-19, Using Consumer Wearable Devices in Health Care Workers: Prospective Model Validation Study.

The early detection of respiratory infections could improve responses against outbreaks. Wearable devices can provide insights into health and well-being using longitudinal physiological signals. The purpose of this study was to prospectively evaluate the performance of a consumer wearable physiology-based respiratory infection detection algorithm in health care workers. In this study, we evaluated the performance of a previously developed system to predict the presence of COVID-19 or other upper respiratory infections. The system generates real-time alerts using physiological signals recorded from a smartwatch. Resting heart rate, respiratory rate, and heart rate variability measured during the sleeping period were used for prediction. After baseline recordings, when participants received a notification from the system, they were required to undergo testing at a Northwell Health System site. Participants were asked to self-report any positive tests during the study. The accuracy of model prediction was evaluated using respiratory infection results (laboratory results or self-reports), and postnotification surveys were used to evaluate potential confounding factors. A total of 577 participants from Northwell Health in New York were enrolled in the study between January 6, 2022, and July 20, 2022. Of these, 470 successfully completed the study, 89 did not provide sufficient physiological data to receive any prediction from the model, and 18 dropped out. Out of the 470 participants who completed the study and wore the smartwatch as required for the 16-week study duration, the algorithm generated 665 positive alerts, of which 153 (23.0%) were not acted upon to undergo testing for respiratory viruses. Across the 512 instances of positive alerts that involved a respiratory viral panel test, 63 had confirmed respiratory infection results (ie, COVID-19 or other respiratory infections detected using a polymerase chain reaction or home test) and the remaining 449 had negative upper respiratory infection test results. Across all cases, the estimated false-positive rate based on predictions per day was 2%, and the positive-predictive value ranged from 4% to 10% in this specific population, with an observed incidence rate of 198 cases per week per 100,000. Detailed examination of questionnaires filled out after receiving a positive alert revealed that physical or emotional stress events, such as intense exercise, poor sleep, stress, and excessive alcohol consumption, could cause a false-positive result. The real-time alerting system provides advance warning on respiratory viral infections as well as other physical or emotional stress events that could lead to physiological signal changes. This study showed the potential of wearables with embedded alerting systems to provide information on wellness measures.

A deep learning-based model to estimate pulmonary function from chest x-rays: multi-institutional model development and validation study in Japan

Chest x-ray is a basic, cost-effective, and widely available imaging method that is used for static assessments of organic diseases and anatomical abnormalities, but its ability to estimate dynamic measurements such as pulmonary function is unknown. We aimed to estimate two major pulmonary functions from chest x-rays. In this retrospective model development and validation study, we trained, validated, and externally tested a deep learning-based artificial intelligence (AI) model to estimate forced vital capacity (FVC) and forced expiratory volume in 1 s (FEV1) from chest x-rays. We included consecutively collected results of spirometry and any associated chest x-rays that had been obtained between July 1, 2003, and Dec 31, 2021, from five institutions in Japan (labelled institutions A-E). Eligible x-rays had been acquired within 14 days of spirometry and were labelled with the FVC and FEV1. X-rays from three institutions (A-C) were used for training, validation, and internal testing, with the testing dataset being independent of the training and validation datasets, and then x-rays from the two other institutions (D and E) were used for independent external testing. Performance for estimating FVC and FEV1 was evaluated by calculating the Pearson's correlation coefficient (r), intraclass correlation coefficient (ICC), mean square error (MSE), root mean square error (RMSE), and mean absolute error (MAE) compared with the results of spirometry. We included 141 734 x-ray and spirometry pairs from 81 902 patients from the five institutions. The training, validation, and internal test datasets included 134 307 x-rays from 75 768 patients (37 718 [50%] female, 38 050 [50%] male; mean age 56 years [SD 18]), and the external test datasets included 2137 x-rays from 1861 patients (742 [40%] female, 1119 [60%] male; mean age 65 years [SD 17]) from institution D and 5290 x-rays from 4273 patients (1972 [46%] female, 2301 [54%] male; mean age 63 years [SD 17]) from institution E. External testing for FVC yielded r values of 0·91 (99% CI 0·90-0·92) for institution D and 0·90 (0·89-0·91) for institution E, ICC of 0·91 (99% CI 0·90-0·92) and 0·89 (0·88-0·90), MSE of 0·17 L2 (99% CI 0·15-0·19) and 0·17 L2 (0·16-0·19), RMSE of 0·41 L (99% CI 0·39-0·43) and 0·41 L (0·39-0·43), and MAE of 0·31 L (99% CI 0·29-0·32) and 0·31 L (0·30-0·32). External testing for FEV1 yielded r values of 0·91 (99% CI 0·90-0·92) for institution D and 0·91 (0·90-0·91) for institution E, ICC of 0·90 (99% CI 0·89-0·91) and 0·90 (0·90-0·91), MSE of 0·13 L2 (99% CI 0·12-0·15) and 0·11 L2 (0·10-0·12), RMSE of 0·37 L (99% CI 0·35-0·38) and 0·33 L (0·32-0·35), and MAE of 0·28 L (99% CI 0·27-0·29) and 0·25 L (0·25-0·26). This deep learning model allowed estimation of FVC and FEV1 from chest x-rays, showing high agreement with spirometry. The model offers an alternative to spirometry for assessing pulmonary function, which is especially useful for patients who are unable to undergo spirometry, and might enhance the customisation of CT imaging protocols based on insights gained from chest x-rays, improving the diagnosis and management of lung diseases. Future studies should investigate the performance of this AI model in combination with clinical information to enable more appropriate and targeted use. None.

Predicting negative attitudes towards suicide in social media texts: prediction model development and validation study.

Implementing machine learning prediction of negative attitudes towards suicide may improve health outcomes. However, in previous studies, varied forms of negative attitudes were not adequately considered, and developed models lacked rigorous external validation. By analyzing a large-scale social media dataset (Sina Weibo), this paper aims to fully cover varied forms of negative attitudes and develop a classification model for predicting negative attitudes as a whole, and then to externally validate its performance on population and individual levels. 938,866 Weibo posts with relevant keywords were downloaded, including 737,849 posts updated between 2009 and 2014 (2009-2014 dataset), and 201,017 posts updated between 2015 and 2020 (2015-2020 dataset). (1) For model development, based on 10,000 randomly selected posts from 2009 to 2014 dataset, a human-based content analysis was performed to manually determine labels of each post (non-negative or negative attitudes). Then, a computer-based content analysis was conducted to automatically extract psycholinguistic features from each of the same 10,000 posts. Finally, a classification model for predicting negative attitudes was developed on selected features. (2) For model validation, on the population level, the developed model was implemented on remaining 727,849 posts from 2009 to 2014 dataset, and was externally validated by comparing proportions of negative attitudes between predicted and human-coded results. Besides, on the individual level, similar analyses were performed on 300 randomly selected posts from 2015 to 2020 dataset, and the developed model was externally validated by comparing labels of each post between predicted and actual results. For model development, the F1 and area under ROC curve (AUC) values reached 0.93 and 0.97. For model validation, on the population level, significant differences but very small effect sizes were observed for the whole sample (χ 2 1 = 32.35, p < 0.001; Cramer's V = 0.007, p < 0.001), men (χ 2 1 = 9.48, p = 0.002; Cramer's V = 0.005, p = 0.002), and women (χ 2 1 = 25.34, p < 0.001; Cramer's V = 0.009, p < 0.001). Besides, on the individual level, the F1 and AUC values reached 0.76 and 0.74. This study demonstrates the efficiency and necessity of machine learning prediction of negative attitudes as a whole, and confirms that external validation is essential before implementing prediction models into practice.

Automated cooling tower detection through deep learning for Legionnaires’ disease outbreak investigations: a model development and validation study

Cooling towers containing Legionella spp are a high-risk source of Legionnaires' disease outbreaks. Manually locating cooling towers from aerial imagery during outbreak investigations requires expertise, is labour intensive, and can be prone to errors. We aimed to train a deep learning computer vision model to automatically detect cooling towers that are aerially visible. Between Jan 1 and 31, 2021, we extracted satellite view images of Philadelphia (PN, USA) and New York state (NY, USA) from Google Maps and annotated cooling towers to create training datasets. We augmented training data with synthetic data and model-assisted labelling of additional cities. Using 2051 images containing 7292 cooling towers, we trained a two-stage model using YOLOv5, a model that detects objects in images, and EfficientNet-b5, a model that classifies images. We assessed the primary outcomes of sensitivity and positive predictive value (PPV) of the model against manual labelling on test datasets of 548 images, including from two cities not seen in training (Boston [MA, USA] and Athens [GA, USA]). We compared the search speed of the model with that of manual searching by four epidemiologists. The model identified visible cooling towers with 95·1% sensitivity (95% CI 94·0-96·1) and a PPV of 90·1% (95% CI 90·0-90·2) in New York City and Philadelphia. In Boston, sensitivity was 91·6% (89·2-93·7) and PPV was 80·8% (80·5-81·2). In Athens, sensitivity was 86·9% (75·8-94·2) and PPV was 85·5% (84·2-86·7). For an area of New York City encompassing 45 blocks (0·26 square miles), the model searched more than 600 times faster (7·6 s; 351 potential cooling towers identified) than did human investigators (mean 83·75 min [SD 29·5]; mean 310·8 cooling towers [42·2]). The model could be used to accelerate investigation and source control during outbreaks of Legionnaires' disease through the identification of cooling towers from aerial imagery, potentially preventing additional disease spread. The model has already been used by public health teams for outbreak investigations and to initialise cooling tower registries, which are considered best practice for preventing and responding to outbreaks of Legionnaires' disease. None.

Nomogram to Estimate the Risk of Chronic Kidney Disease-Associated Pruritus in Patients with End-Stage Renal Disease Undergoing Peritoneal Dialysis: Model Development and Validation Study

Plain Language SummaryWe conducted a study to better understand and predict the risk of CKD-associated pruritus, a common itching condition in patients with severe kidney disease undergoing peritoneal dialysis. Our research included an analysis of the risk factors associated with this condition. Using information from uremic pruritus patients, we developed a special tool that takes into account factors such as a patient’s age, certain blood markers, hemoglobin levels, residual urine volume, and renal Kt/V. This tool is designed to help doctors estimate a patient’s likelihood of developing CKD-associated pruritus. Our results show that this tool is not only accurate but also practical for use in medical settings, aiding doctors in making informed treatment decisions.