Deep learning algorithm performance compared to experts in visual evaluation of interior vena cava collapse on ultrasound to determine intravenous fluid need in dehydration management

Abstract

Objectives: To create a deep learning (DL) algorithm capable of analyzing real time ultrasound video of the inferior vena cava (IVC) for complete collapse in pediatric patients being evaluated for intravenous fluid (IVF) resuscitation. Methods: Researchers employed a VGG-16 based DL architecture, running inside a Long Short Term Memory algorithm design, to analyze prospectively obtained ultrasound video from pediatric patients presenting with dehydration to a busy urban ED, obtained for a prior clinical study. All videos were de-identified and no patient information was available. A total of 184 patient IVC ultrasound videos were used in the study. All videos were previously reviewed and graded by two blinded POCUS experts (PedEM fellow and PedEM attending with 20 years experience) and split into two categories, those showing complete (95 patients) and those with incomplete (89 patients) IVC collapse. Approximately 10% (9) patient videos were randomly removed from each original data groups to be used for algorithm testing after training was completed. A standard 80%/20% training and validation split was used on the remaining 166 patient videos for algorithm training. Training accuracy, losses and learning curves were tracked and various training parameters such as learning rates and batch sizes were optimized throughout training. As a final real world test, the DL algorithm was tasked with analyzing the 18 previously unseen, randomly selected IVC videos. Cohen’s kappa was calculated for each of the blinded POCUS reviewers and DL algorithm. Results: This DL algorithm completed analysis of each previously unseen real world test video and is the first such algorithm to analyze IVC collapse through visual estimation in real-time. The algorithm was able to deliver a collapse result prediction for all 18 test IVC videos and there were no failures. Algorithm agreement with PedEM POCUS attending was substantial with a Cohen’s kappa of 0.78 (95% CI 0.49 to 1.0). Algorithm agreement with PedEM Fellow was substantial with Cohen’s kappa of 0.66 (95% CI 0.31 to 1.0). The PEM fellow and PEM POCUS attending also had substantial agreement, yielding a Cohen’s kappa of 0.66 (95% CI 0.32 to 1.0). Conclusions: This DL algorithm developed on prospectively acquired IVC video data from patients being studied for an IVF resuscitation study proved accurate at identifying when the IVC collapsed completely in real time. There was substantial agreement with POCUS reviewers of the same videos. Such an algorithm could allow novice clinicians to rapidly identify complete IVC collapse in children and the need for IVF administration. This could expand patient access to point of care technology by enabling novices with little training to use the diagnostic tool at bedside and decide if patients require intravenous fluid administration.