Semi-Supervised Action Recognition From Newborn Resuscitation Videos

Birth asphyxia is one of the leading causes of neonatal deaths. A key for survival is performing immediate and continuous quality newborn resuscitation. A dataset of recorded signals during newborn resuscitation, including videos, has been collected in Haydom, Tanzania, and the aim is to analyze the treatment and its effect on the newborn outcome. An important step is to generate timelines of relevant resuscitation activities, including ventilation, stimulation, suction, etc., during the resuscitation episodes. We propose a two-step deep neural network system, ORAA-net, utilizing low-quality video recordings of resuscitation episodes to do activity recognition during newborn resuscitation. The first step is to detect and track relevant objects using Convolutional Neural Networks (CNN) and post-processing, and the second step is to analyze the proposed activity regions from step 1 to do activity recognition using 3D CNNs. The system recognized the activities newborn uncovered, stimulation, ventilation and suction with a mean precision of 77.67%, a mean recall of 77,64%, and a mean accuracy of 92.40%. Moreover, the accuracy of the estimated number of Health Care Providers (HCPs) present during the resuscitation episodes was 68.32%. The results indicate that the proposed CNN-based two-step ORAA-net could be used for object detection and activity recognition in noisy low-quality newborn resuscitation videos. A thorough analysis of the effect the different resuscitation activities have on the newborn outcome could potentially allow us to optimize treatment guidelines, training, debriefing, and local quality improvement in newborn resuscitation.

OBJECTIVES: To evaluate the current practice variabilities during resuscitation of preterm infants by the dedicated Resuscitation Stabilization Team (RST) using videos and respiratory function recordings of the delivery room management. DESIGN/METHODS: At our center, neonatal stabilization rooms are equipped with video recording and respiratory function monitor. We analyzed the first 10 minutes of delivery room stabilization of preterm infants at birth. The RST performance was evaluated and compared against the Canadian and regional Neonatal Resuscitation Guidelines. RESULTS: Thirty infants were video recorded over 8 months, with mean gestational age (GA) 26 (±2) weeks and birth weight 960 (±315)g. There was 100% compliance with using the plastic drape for infants less than 28 weeks GA. EKG leads and Pulse Oximetry were applied to all 30 patients. The median time[IQR] for application of the pulse oximetry was 47 seconds [35- 65] from the time of arrival at the table. Only 9/30 infants were suctioned prior to starting the respiratory support. There were inconsistencies in drying and stimulation within first minute for infants less than 28 weeks GA. There was a trend of initiating mask Continuous Positive Airway Pressure (CPAP) prior to completing initial assessment for adequacy of spontaneous breathing. 14 infants were apneic when placed on the table. The median [IQR] time to initiate positive pressure ventilation (PPV) in these apneic babies was 26 seconds [12-37.5]. 5/9 apneic babies didn’t have clinical assessment of heart rate as a part of initial assessment or to establish effectiveness of the ventilation. There were 10 events in 7 patients where PPV was interrupted by the PPV provider for purposes other than ventilation corrective steps. Early initiation of nasal CPAP i.e., less than 10 minutes was noted in 8 babies. CONCLUSION: The results of the RST performance are comparable to available literature. The results represent efficient neonatal stabilizations by a well-trained stabilization team. The variability of sequence in accomplishing each step of resuscitation could indicate resuscitator’s training and experience for individual skill set or judgement on rapidly changing clinical situation. There is a need of ongoing resuscitation training with special focus on situational awareness to prepare the NRP providers for timely strategized performance. Resuscitation videos can be a useful tool for educational and training of NRP providers.The results of the RST performance are comparable to available literature. The results represent efficient neonatal stabilizations by a well-trained stabilization team. The variability of sequence in accomplishing each step of resuscitation could indicate resuscitator’s training and experience for individual skill set or judgement on rapidly changing clinical situation. There is a need of ongoing resuscitation training with special focus on situational awareness to prepare the NRP providers for timely strategized performance. Resuscitation videos can be a useful tool for educational and training of NRP providers.The results of the RST performance are comparable to available literature. The results represent efficient neonatal stabilizations by a well-trained stabilization team. The variability of sequence in accomplishing each step of resuscitation could indicate resuscitator’s training and experience for individual skill set or judgement on rapidly changing clinical situation. There is a need of ongoing resuscitation training with special focus on situational awareness to prepare the NRP providers for timely strategized performance. Resuscitation videos can be a useful tool for educational and training of NRP providers.

Approximately 3% of births require neonatal resuscitation, which has a direct impact on the immediate survival of these infants. This report proposes an automatic video analysis method for neonatal resuscitation performance evaluation, which helps improve the quality of this procedure. More specifically, we design a deep learning based action model which incorporates motion and spatial information in order to classify neonatal resuscitation actions in videos. First, we use a Convolutional Neural Network to select regions containing infants and only keep those that are motion salient. Second, we extract deep spatial-temporal features to train a linear SVM classifier. Finally, we propose a pair-wise model to ensure consistent classification in consecutive frames. We evaluate the proposed method on a dataset consisting of 17 videos and compare the result against the state-of-the-art method for action classification in videos. To our best knowledge, this work is the first to attempt automatic evaluation of neonatal resuscitation videos and identifies several issues that require further work.

The quality of neonatal resuscitation after delivery needs to be improved to reach the Sustainable Development Goals 3.2 (reducing neonatal deaths to <12/1,000 live newborns) by the year 2030. Studies have emphasized the importance of correctly performing the basic steps of resuscitation including stimulation, heart rate assessment, ventilation, and thermal control. Recordings with video cameras have previously been shown to be one way to identify performance practices during neonatal resuscitation. A description of a low-cost delivery room set up for video recording of neonatal resuscitation. The technical setup includes rechargeable high-definition cameras with two-way audio, NeoBeat heart rate monitors, and the NeoTapAS data collection tools for iPad with direct data export of data for statistical analysis. The setup was field tested at Mulago National Referral Hospital, Kampala, Uganda, and Phu San Hanoi Hospital, Hanoi, Vietnam. The setup provided highly detailed resuscitation video footage including data on procedures and team performance, heart rate monitoring, and clinical assessment of the neonate. The data were analyzed with the free-of-charge NeoTapAS for iPad, which allowed fast and accurate registration of all resuscitative events. All events were automatically registered and exported to R statistical software for further analysis. Video analysis of neonatal resuscitation is an emerging quality assurance tool with the potential to improve neonatal resuscitation outcomes. Our methodology and technical setup are well adapted for low- and lower-middle-income countries settings where improving neonatal resuscitation outcomes is crucial. This delivery room video recording setup also included two-way audio communication that potentially could be implemented in day-to-day practice or used with remote teleconsultants.

Neonatal resuscitation video review (NRVR) involves recording and reviewing resuscitations for education and quality assurance. Though NRVR has been shown to improve teamwork and skill retention, it is not widely used. We evaluated clinicians' experiences of NRVR to understand how NRVR impacts learning and can be improved. Neonatal Intensive Care Unit (NICU) clinicians with previous NRVR experience were recruited for individual semi-structured interviews. Using a social constructivist viewpoint, five researchers used thematic analysis to analyze participant responses. Twenty-two clinicians (11 nurses, 11 doctors) were interviewed. All participants expressed positive attitudes towards NRVR. Four themes were identified: (1) Learning from reality-exposure to real-life resuscitations was highly clinically relevant. (2) Immersive self-regulation-watching videos aided recall and reflection. (3) Complexities in learner psychological safety-all participants acknowledged viewing NRVR videos could be confronting. Some expressed fear of judgment from colleagues, though the educational benefit of NRVR superseded this. (4) Accessing and learning from diverse vantage points-NRVR promoted group discussion, which prompted participant learning from colleagues' viewpoints. Neonatal clinicians reported NRVR to be an effective and safe method for learning and refining skills required during neonatal resuscitation, such as situational awareness and communication. Neonatal resuscitation video review is not known to be widely used in neonatal resuscitation teaching, and published research in this area is limited. Our study examined clinician attitudes towards an established neonatal resuscitation video review program. We found strong support for teaching using neonatal resuscitation video review among neonatal doctors and nurses, with key benefits including increased situational awareness and increased clinical exposure to resuscitations, while maintaining psychological safety for participants. The results of this study add evidence to support the addition of video review to neonatal resuscitation training.

Birth asphyxia is a major newborn mortality problem in low-resource countries. International guideline provides treatment recommendations; however, the importance and effect of the different treatments are not fully explored. The available data are collected in Tanzania, during newborn resuscitation, for analysis of the resuscitation activities and the response of the newborn. An important step in the analysis is to create activity timelines of the episodes, where activities include ventilation, suction, stimulation, etc. Methods: The available recordings are noisy real-world videos with large variations. We propose a two-step process in order to detect activities possibly overlapping in time. The first step is to detect and track the relevant objects, such as bag-mask resuscitator, heart rate sensors, etc., and the second step is to use this information to recognize the resuscitation activities. The topic of this paper is the first step, and the object detection and tracking are based on convolutional neural networks followed by post processing. The performance of the object detection during activities were 96.97% (ventilations), 100% (attaching/removing heart rate sensor), and 75% (suction) on a test set of 20 videos. The system also estimate the number of health care providers present with a performance of 71.16%. The proposed object detection and tracking system provides promising results in noisy newborn resuscitation videos. This is the first step in a thorough analysis of newborn resuscitation episodes, which could provide important insight about the importance and effect of different newborn resuscitation activities.

Assessment of temporal variations in adherence to NRP using video recording in the delivery room

Newborn mortality is a global challenge with around 2.4 million neonatal deaths in 2019. One third of these occur within the first-and-only day of life with labour complications and birth asphyxia being the primary causes. Existing guidelines for newborn resuscitation are based on limited scientific evidence, and evidens based research is sought for. To increase our knowledge on resuscitation of newborns, it is crucial to first quantify what is currently being done in terms of therapeutic activities, such as ventilation and stimulation, and how they affect resuscitation outcomes. In the current study, the therapeutic activities during newborn resuscitation are quantified by estimating a timeline describing the start and stop of activities. The proposed approach is combining methods using both video and time series data recorded during resuscitation, where the predictions are based on the available sources. From video the activity recognition is done by a 3D CNN method. For the signal data feature extraction is performed on ECG and accelerometer signals and thereafter machine learning is done to perform stimulation detection. We show that best results are achieved with all signals and video available, for the activity “stimulation” we get an AUC of 0.86, sensitivity of 82.32%, specificity of 82.23%, and precision of 57.59%. If only signals or video is available we still get good results with AUC at 0.80, and 0.84 respectively.

BackgroundApproximately 3-8% of all newborns do not breathe spontaneously at birth, and require time critical resuscitation. Resuscitation guidelines are mostly based on best practice, and more research on newborn resucitation is highly sought for.MethodsThe NewbornTime project will develop artificial intelligence (AI) based solutions for activity recognition during newborn resuscitations based on both visible light spectrum videos and infrared spectrum (thermal) videos. In addition, time-of-birth detection will be developed using thermal videos from the delivery rooms. Deep Neural Network models will be developed, focusing on methods for limited supervision and solutions adapting to on-site environments. A timeline description of the video analysis output enables objective analysis of resuscitation events. The project further aims to use machine learning to find patterns in large amount of such timeline data to better understand how newborn resuscitation treatment is given and how it can be improved. The automatic video analysis and timeline generation will be developed for on-site usage, allowing for data-driven simulation and clinical debrief for health-care providers, and paving the way for automated real-time feedback. This brings added value to the medical staff, mothers and newborns, and society at large.DiscussionThe project is a interdisciplinary collaboration, combining AI, image processing, blockchain and cloud technology, with medical expertise, which will lead to increased competences and capacities in these various fields.Trial registrationISRCTNregistry, number ISRCTN12236970

Neonatal resuscitation video review - has the time for wider adoption come?