#1017 External validation of a machine learning model for the prediction of bioimpedance-determined pre-dialysis overhydration

Abstract

Abstract Background and Aims An accurate, automated determination of volume status in dialysis patients using a reproducible standard would be clinically useful. We have previously developed a machine learning model for the prediction of pre-dialysis overhydration (OH) index (litres) from the Fresenius Body Composition Monitor (BCM). External validation is a requisite step before the clinical implementation of a machine learning model. Method Data analyses were performed using Python. An XGBoost regressor model was internally developed and validated in a Fresenius dataset of 9,452 patients and 279,324 BCM target events from the Czech Republic and Italy between January 2016 and December 2022. An XGBoost regressor model with 60 input variables including the average BCM OH value in the preceding year, 2-week moving averages of dialysis session data and biomarkers/ medications data from the preceding 13 weeks demonstrated the best performance of the models tested. The external validation (EV) dataset consisted of 24 patients from a 7-week observational study in a single centre, in Beaumont, Ireland comprising 94 BCM sessions and 650 haemodialysis sessions. Individualised predictions were compared with BCM measures and clinical volume assessments performed at 2-week intervals. Mean squared error (MSE), Root mean square error (RMSE), and Bland-Altman plots were used for continuous outcomes. Precision, recall and F1 score were used to assess the algorithm's performance in predicting the following fluid categories: overhydration ≥1.1L, normohydration −1.1L to 1.1L, underhydration ≤-1.1L. Normohydration weight was calculated by subtracting the predicted OH index from the pre-dialysis weight. Results There was performance degradation between the internal validation (IV) and EV set. The RMSE for pre-dialysis OH was 1.04 kg in the IV set and 1.5 kg in the EV set. This could be attributed to differences in the distribution of the outcome variable (BCM OH validation dataset=1.97 ± 1.56 kg*, EV dataset=0.89 ± 1.7 kg), and the case-mix. The RMSE for normohydration weight was 1.86 kg in the EV set. Wide limits of agreement were noted on a Bland Altman plot comparing the average of BCM and predicted values plotted against the difference in values (mean bias - 0.54, 95% limits of agreement -3.44 kg to 2.36 kg). The overall accuracy of the model in predicting fluid classes was 60%. A histogram showed a close alignment in the distributions of predicted normohydration weights and BCM values. Conclusion The performance of the algorithms degraded between the internal validation and external validation datasets. However, the pre-dialysis algorithm showed an ability to discriminate fluid categories compared to nursing staff and the overall accuracy was acceptable. Additional training of the algorithms using fluid status classes would be expected to improve the precision of predictions. *Values less than the 1st percentile of – 2.17 L were excluded.