Video Annotation Research Articles

Artificial Intelligence Vision Based Analysis and Key Steps Identification Feasibility Study in Laparoscopic Hysterectomies

<h3>Study Objective</h3> The analysis of surgical videos and key steps identification using artificial intelligence (AI) holds great promise for the future of healthcare. In this paper, we present a novel computer vision algorithm for surgical steps identification in laparoscopic hysterectomy. <h3>Design</h3> A retrospective analysis of surgical videos of laparoscopic hysterectomies. <h3>Setting</h3> Gynecological department in a tertiary hospital with an average of 6500 procedures a year. <h3>Patients or Participants</h3> 190 laparoscopic hysterectomies from September 2020 to April 2022. <h3>Interventions</h3> Artificial intelligence-driven surgical platform that uses advanced computer vision technology to capture video data during all surgeries, de-identify it, and upload it to a secure cloud infrastructure. <h3>Measurements and Main Results</h3> A total of 190 full-length hysterectomies videos were manually annotated with sequential steps of surgery. Of these, 115 cases served as a training dataset for algorithm development, 28 cases were used for internal validation, and 47 were used as a separate testing cohort for evaluating algorithm accuracy. Concordance between AI-enabled automated video analysis and manual human video annotation was 92%. Algorithm accuracy was highest for the vaginal vault closure step (97%) and lowest for the adhesiolysis step (71%). <h3>Conclusion</h3> A variety of AI applications are expanding in clinical systems, from databases to intraoperative video analysis. The unique nature of surgical practice puts surgeons in a strong position to contribute to the next phase of AI, focused on real-time clinical decision support designed to optimize surgeon workflow and patient care. This work validates the ability to annotate the different steps of hysterectomy using an AI vision-based algorithm. Automated surgical video analysis has immediate impact in this field and the implementation of such a system holds enormous promise for future surgical education and improvement.

Fast Weakly Supervised Action Segmentation Using Mutual Consistency.

Action segmentation is the task of predicting the actions for each frame of a video. As obtaining the full annotation of videos for action segmentation is expensive, weakly supervised approaches that can learn only from transcripts are appealing. In this paper, we propose a novel end-to-end approach for weakly supervised action segmentation based on a two-branch neural network. The two branches of our network predict two redundant but different representations for action segmentation and we propose a novel mutual consistency (MuCon) loss that enforces the consistency of the two redundant representations. Using the MuCon loss together with a loss for transcript prediction, our proposed approach achieves the accuracy of state-of-the-art approaches while being 14 times faster to train and 20 times faster during inference. The MuCon loss proves beneficial even in the fully supervised setting.

The Arousal Video Game AnnotatIoN (AGAIN) Dataset

How can we model affect in a general fashion, across dissimilar tasks, and to which degree are such general representations of affect even possible? To address such questions and enable research towards general affective computing, this paper introduces The Arousal video Game AnnotatIoN (AGAIN) dataset. AGAIN is a large-scale affective corpus that features over 1,100 in-game videos (with corresponding gameplay data) from nine different games, which are annotated for arousal from 124 participants in a first-person continuous fashion. Even though AGAIN is created for the purpose of investigating the generality of affective computing across dissimilar tasks, affect modelling can be studied within each of its 9 specific interactive games. To the best of our knowledge AGAIN is the largest -- over 37 hours of annotated video and game logs -- and most diverse publicly available affective dataset based on games as interactive affect elicitors.

Detecting Marine Organisms Via Joint Attention-Relation Learning for Marine Video Surveillance

The better way to understand marine life and ecosystems is to surveil and analyze the activities of marine organisms. Recently, research on marine video surveillance is becoming increasingly popular. With the rapid development of deep learning (DL), convolutional neural networks (CNNs) have made remarkable progresses in image/video understanding tasks. In this article, we explore a visual attention and relation mechanism for marine organism detection, and propose a new way to apply an improved attention-relation (AR) module on an efficient marine organism detector (EMOD), which can well enhance the discrimination of organisms in complex underwater environments. We design our EMOD via integrating current state-of-the-art (SOTA) detection methods in order to detect organisms and surveil marine environments in a real time and fast fashion for high-resolution marine video surveillance. We implement our EMOD and AR on the annotated video data sets provided by the public data challenges in conjunction with the workshops (CVPR 2018 and 2019), which are supported by National Oceanic and Atmospheric Administration (NOAA) and their research works (NMFS-PIFSC-83). Experimental results and visualizations demonstrate that our application of AR module is effective and efficient, and our EMOD equipped with AR modules can outperform SOTA performance on the experimental data sets. For application requirements, we also provide the application suggestions of EMOD framework. Our code is publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/zhenglab/EMOD</uri> .

The Ziegler Test is Reliable and Valid for Measuring Freezing of Gait in People With Parkinson Disease.

The purpose of this study was to determine interrater and test-retest reliability of the Ziegler test to measure freezing of gait (FOG) severity in people with Parkinson disease. Secondary aims were to evaluate test validity and explore Ziegler test duration as a proxy FOG severity measure. Physical therapists watched 36 videos of people with Parkinson disease and FOG perform the Ziegler test and rated FOG severity using the rating scale in real time. Two researchers rated 12 additional videos and repeated the ratings at least 1week later. Interrater and test-retest reliability were calculated using intraclass correlation coefficients (ICCs). Bland-Altman plots were used to visualize agreement between the researchers for test-retest reliability. Correlations between the Ziegler scores, Ziegler test duration, and percentage of time frozen (based on video annotations) were determined using Pearson r. Twenty-four physical therapists participated. Overall, the Ziegler test showed good interrater (ICC2,1 = 0.80; 95% CI = 0.65 to 0.92) and excellent test-retest (ICC3,1 = 0.91; 95% CI = 0.82 to 0.96) reliability when used to measure FOG. It was also a valid measure, with a high correlation (r = 0.72) between the scores and percentage of time frozen. Ziegler test duration was moderately correlated (r = 0.67) to percentage of time frozen and may be considered a proxy FOG severity measure. The Ziegler test is a reliable and valid tool to measure FOG when used by physical therapists in real time. Ziegler test duration may be used as a proxy for measuring FOG severity.

The Gray Area of Freezing of Gait Annotation: A Guideline and Open-Source Practical Tool.

Freezing of gait, a disabling episodic symptom, is difficult to assess as the exact begin- and endpoint of an episode is not easy to specify. This hampers scientific and clinical progress. The current golden standard is video annotation by two independent raters. However, the comparison of the two ratings gives rise to non-overlapping, gray areas. To provide a guideline for dealing with these gray areas. We propose a standardized procedure for handling the gray areas based on two parameters, the tolerance and correction parameter. Furthermore, we recommend the use of positive agreement, negative agreement, and prevalence index to report interrater agreement instead of the commonly used intraclass correlation coefficient or Cohen's kappa. This theoretical guideline was implemented in an open-source practical tool, FOGtool (https://github.com/helenacockx/FOGtool). This paper aims to contribute to the standardization of freezing of gait assessment, thereby improving data sharing procedures and replicability of study results.

Saliency-assisted multi-label classification for explainable deep learning applications in endoscopic ENT navigation

Abstract Introduction: In endoscopic procedures of the nasal sinuses, a critical issue for classification tasks is the ambiguous anatomical representations due to their complex composition. We investigated the potential of multi-label image-based classifications of sinus landmark combinations together with explainability methods for machine learning in an assistance function at the application level. By combining image classification and pixel attribution in a navigation function, we provide the surgeon with label predictions and additional localization cues of important pixels relevant to the model output with regard to the input image. Methods: We used 3500 label annotated video sequences from 30 recorded sinus surgeries to fine-tune a pretrained ResNet50 as the feature extractor and a classification head using binary cross-entropy on one-hot encoded target vectors of landmark classes with a RAdam optimizer over 28-32 epochs. Image augmentation and a focal loss function were added to counter over-fitting. An explainability function used the trained model to produce pixel attribution maps for predicted classes regarding individual input images. These gradient maps were summed over all classes, and pixel values above 0.20 were clustered using weighted k-means based on the gradient value of each pixel coordinate. The resulting cluster centroids were then overlapped into the endoscopic image with the predicted landmark classes. Three surgeons investigated three different overlay scenarios in a validation study. Results: The top-1 predictions reached a mean f1-score of 0.47, with the highest values of 0.71 and the lowest with 0.28. Despite over-fitting mitigation, prediction results largely depended on over- and underrepresented classes. Conclusion: The provided explainability function at the application level showed the strong potential of delivering visual cues from prediction results to surgeons at runtime to support human-machine interaction.

Using High-Fidelity Avatars to Advance Camera-Based Cardiac Pulse Measurement.

Non-contact physiological measurement has the potential to provide low-cost, non-invasive health monitoring. However, machine vision approaches are often limited by the availability and diversity of annotated video datasets resulting in poor generalization to complex real-life conditions. To address these challenges, this work proposes the use of synthetic avatars that display facial blood flow changes and allow for systematic generation of samples under a wide variety of conditions. Our results show that training on both simulated and real video data can lead to performance gains under challenging conditions. We show strong performance on three large benchmark datasets and improved robustness to skin type and motion. These results highlight the promise of synthetic data for training camera-based pulse measurement; however, further research and validation is needed to establish whether synthetic data alone could be sufficient for training models.

Do Individual Surgeon Preferences Affect Procedural Outcomes?

Surgeon preferences such as instrument and suture selection and idiosyncratic approaches to individual procedure steps have been largely viewed as minor differences in the surgical workflow. We hypothesized that idiosyncratic approaches could be quantified and shown to have measurable effects on procedural outcomes. At the American College of Surgeons (ACS) Clinical Congress, experienced surgeons volunteered to wear motion tracking sensors and be videotaped while evaluating a loop of porcine intestines to identify and repair 2 preconfigured, standardized enterotomies. Video annotation was used to identify individual surgeon preferences and motion data was used to quantify surgical actions. χ 2 analysis was used to determine whether surgical preferences were associated with procedure outcomes (bowel leak). Surgeons' (N=255) preferences were categorized into 4 technical decisions. Three out of the 4 technical decisions (repaired injuries together, double-layer closure, corner-stitches vs no corner-stitches) played a significant role in outcomes, P <0.05. Running versus interrupted did not affect outcomes. Motion analysis revealed significant differences in average operative times (leak: 6.67min vs no leak: 8.88min, P =0.0004) and work effort (leak-path length=36.86cm vs no leak-path length=49.99cm, P =0.001). Surgeons who took the riskiest path but did not leak had better bimanual dexterity (leak=0.21/1.0 vs no leak=0.33/1.0, P =0.047) and placed more sutures during the repair (leak=4.69 sutures vs no leak=6.09 sutures, P =0.03). Our results show that individual preferences affect technical decisions and play a significant role in procedural outcomes. Future analysis in more complex procedures may make major contributions to our understanding of contributors to procedure outcomes.

One year of general practice during the COVID-19 pandemic - presentation and evaluation of digital medical education.

Background and teaching situation: The SARS-CoV-2 pandemic had a substantial didactic impact on medical teaching. In Erlangen, the lecture “General Practice” was offered asynchronously and digitally in an inverted-classroom concept. Contents were available via a learning platform. The lecture was presented using annotated videos, consolidation materials and control questions. A forum encouraged for discussions and feedback and collected in-depth aspects for a case-based video consultation. The aim of this work is to evaluate and critically examine the digital teaching concept during the SARS-CoV-2 pandemic.Methodology: Two semester cohorts evaluated the lecture. Overall impression of the lecture, didactic elements, suitability and the desired future lecture format were surveyed quantitatively. Free text answers were evaluated by means of qualitative content synthesis.Results: In terms of overall impression, the students (N=199) rated the lecture on average as “very good” (M=1.41, SD=.57). Digital methods were perceived as suitable for supporting self-study, and digital usage was rated as unproblematically (M=1.18, SD=.50). Desired future teaching formats were blended learning concepts (79.4%). Organisation, structure and content presentation were highly appreciated. The time for completing the course was perceived critically. The students urged for more practical and consolidating lecture work.Discussion and implications: The results illustrate high acceptance of digital teaching and underline the demand for future blended learning concepts. It is particularly important to better consider the students’ time investment and practical relevance of digital self-learning mechanisms.

A Comparison of Video-based Methods for Neonatal Body Motion Detection.

Preterm infants in a neonatal intensive care unit (NICU) are continuously monitored for their vital signs, such as heart rate and oxygen saturation. Body motion patterns are documented intermittently by clinical observations. Changing motion patterns in preterm infants are associated with maturation and clinical events such as late-onset sepsis and seizures. However, continuous motion monitoring in the NICU setting is not yet performed. Video-based motion monitoring is a promising method due to its non-contact nature and therefore unobtrusiveness. This study aims to determine the feasibility of simple video-based methods for infant body motion detection. We investigated and compared four methods to detect the motion in videos of infants, using two datasets acquired with different types of cameras. The thermal dataset contains 32 hours of annotated videos from 13 infants in open beds. The RGB dataset contains 9 hours of annotated videos from 5 infants in incubators. The compared methods include background substruction (BS), sparse optical flow (SOF), dense optical flow (DOF), and oriented FAST and rotated BRIEF (ORB). The detection performance and computation time were evaluated by the area under receiver operating curves (AUC) and run time. We conducted experiments to detect motion and gross motion respectively. In the thermal dataset, the best performance of both experiments is achieved by BS with mean (standard deviation) AUCs of 0.86 (0.03) and 0.93 (0.03). In the RGB dataset, SOF outperforms the other methods in both experiments with AUCs of 0.82 (0.10) and 0.91 (0.05). All methods are efficient to be integrated into a camera system when using low-resolution thermal cameras.

Student Teachers’ Video-Assisted Collaborative Reflections of Socio-Emotional Experiences During Teaching Practicum

This study explores video as a tool for student teachers in reflecting upon their own teaching practice. The particular interest is in exploring, what kind of socio-emotional experiences did the student teachers describe during the video-assisted collaborative reflection and what benefits student teachers experienced through the video-assisted collaborative reflections? The study implements a video-observation model (Participatory and Empowering Video Analysis- model, PEVA™) developed for a professional teacher education programme and investigates the model from student teachers’ perspective. The participants are student teachers in an international professional teacher education programme (n = 24) at a university of applied sciences. Student teachers video recorded their own teaching sessions displaying their work in different roles: teacher-led content specific instructions, group working sessions and individual student guidance and tutoring sessions. Student teachers were advised to annotate their own videos with a video-reflection tool by focussing on socio-emotional interactions, non-verbal elements of interaction and moments, where students felt successful and empowered. In addition to self-reflection, videos were viewed and annotated by a peer student and a teacher educator. After the phases of reviews and annotations (self-,peer-, and teacher annotations), an hour-long collaborative reflective feedback discussion was held either face-to-face or online. These sessions were facilitated by the teacher educator, involving the videoed student and the reviewing peer. Video annotation data was transcribed, and, in addition, the student teachers’ reflective writings (54 pages) and teacher educators’ notes of the reflective feedback sessions (20 pages) were used as data. The results of this study indicate that student teachers viewed the video reflection process as helpful in making socio-emotional experiences visible and tangible. The video annotation tool was recognised as useful for indicating socio-emotional experiences and making them more concrete thus resources for reflection. A combination of own, peer’s and teachers’ annotations functioned as a collaborative tool for increasing awareness of different socio-emotional experiences. By observing videos of their own teaching as well as teaching videos of their peers, student teachers were able to evaluate teaching situations holistically and observe connections of their own socio-emotional behaviours to their students.

Spatial and temporal constrained ranked retrieval over videos

Recent advances in Computer Vision (CV) algorithms have improved accuracy and efficiency, making video annotations possible with high accuracy. In this paper, we utilize the annotated data provided by such algorithms and construct graph representations to capture both object labels and spatial-temporal relationships of objects in videos. We define the problem of Spatial and Temporal Constrained Ranked Retrieval (STAR Retrieval) over videos. Based on the graph representation, we propose a two-phase approach, consisting of the ingestion phase, where we construct and materialize the Graph Index (GI), and the query phase, where we compute the top ranked windows (video clips) according to the window matching score efficiently. We propose two algorithms to perform Spatial Matching (SMA) and Temporal Matching (TM) separately with an early-stopping mechanism. Our experiments demonstrate the effectiveness of the proposed methods, achieving orders of magnitude speedups on queries with high selectivity.

Video annotation with metadata: A case study of soccer game video annotation with players

In recent years, the use of convolutional neural networks for video processing has become very attractive. The reason lies in the computational power for data processing which is available today. There are many well-defined research areas where neural networks have brought higher reliability than other conventional approaches; for example, traffic sign recognition and isolated number recognition. In this paper, we will describe the architecture and the implementation of the process of soccer game annotation. The game is annotated with data about players. The technology of convolutional neural networks is used for number recognition. The process runs in real-time on a streaming video. Content enriched with metadata is given to the user in parallel with the real-time video. In the paper, we will describe in some detail the following modules: Image binarization, shot localization, the selection and recognition of numbers on players` jerseys.

Tracker/Camera Calibration for Accurate Automatic Gaze Annotation of Images and Videos.

Modern appearance-based gaze tracking algorithms require vast amounts of training data, with images of a viewer annotated with "ground truth" gaze direction. The standard approach to obtain gaze annotations is to ask subjects to fixate at specific known locations, then use a head model to determine the location of "origin of gaze". We propose using an IR gaze tracker to generate gaze annotations in natural settings that do not require the fixation of target points. This requires prior geometric calibration of the IR gaze tracker with the camera, such that the data produced by the IR tracker can be expressed in the camera's reference frame. This contribution introduces a simple tracker/camera calibration procedure based on the PnP algorithm and demonstrates its use to obtain a full characterization of gaze direction that can be used for ground truth annotation.

Validation of Machine Learning-Based Automated Surgical Instrument Annotation Using Publicly Available Intraoperative Video.

Intraoperative tool movement data have been demonstrated to be clinically useful in quantifying surgical performance. However, collecting this information from intraoperative video requires laborious hand annotation. The ability to automatically annotate tools in surgical video would advance surgical data science by eliminating a time-intensive step in research. To identify whether machine learning (ML) can automatically identify surgical instruments contained within neurosurgical video. A ML model which automatically identifies surgical instruments in frame was developed and trained on multiple publicly available surgical video data sets with instrument location annotations. A total of 39 693 frames from 4 data sets were used (endoscopic endonasal surgery [EEA] [30 015 frames], cataract surgery [4670], laparoscopic cholecystectomy [2532], and microscope-assisted brain/spine tumor removal [2476]). A second model trained only on EEA video was also developed. Intraoperative EEA videos from YouTube were used for test data (3 videos, 1239 frames). The YouTube data set contained 2169 total instruments. Mean average precision (mAP) for instrument detection on the YouTube data set was 0.74. The mAP for each individual video was 0.65, 0.74, and 0.89. The second model trained only on EEA video also had an overall mAP of 0.74 (0.62, 0.84, and 0.88 for individual videos). Development costs were $130 for manual video annotation and under $100 for computation. Surgical instruments contained within endoscopic endonasal intraoperative video can be detected using a fully automated ML model. The addition of disparate surgical data sets did not improve model performance, although these data sets may improve generalizability of the model in other use cases.

Fast machine learning annotation in the medical domain: a semi-automated video annotation tool for gastroenterologists

BackgroundMachine learning, especially deep learning, is becoming more and more relevant in research and development in the medical domain. For all the supervised deep learning applications, data is the most critical factor in securing successful implementation and sustaining the progress of the machine learning model. Especially gastroenterological data, which often involves endoscopic videos, are cumbersome to annotate. Domain experts are needed to interpret and annotate the videos. To support those domain experts, we generated a framework. With this framework, instead of annotating every frame in the video sequence, experts are just performing key annotations at the beginning and the end of sequences with pathologies, e.g., visible polyps. Subsequently, non-expert annotators supported by machine learning add the missing annotations for the frames in-between.MethodsIn our framework, an expert reviews the video and annotates a few video frames to verify the object’s annotations for the non-expert. In a second step, a non-expert has visual confirmation of the given object and can annotate all following and preceding frames with AI assistance. After the expert has finished, relevant frames will be selected and passed on to an AI model. This information allows the AI model to detect and mark the desired object on all following and preceding frames with an annotation. Therefore, the non-expert can adjust and modify the AI predictions and export the results, which can then be used to train the AI model.ResultsUsing this framework, we were able to reduce workload of domain experts on average by a factor of 20 on our data. This is primarily due to the structure of the framework, which is designed to minimize the workload of the domain expert. Pairing this framework with a state-of-the-art semi-automated AI model enhances the annotation speed further. Through a prospective study with 10 participants, we show that semi-automated annotation using our tool doubles the annotation speed of non-expert annotators compared to a well-known state-of-the-art annotation tool.ConclusionIn summary, we introduce a framework for fast expert annotation for gastroenterologists, which reduces the workload of the domain expert considerably while maintaining a very high annotation quality. The framework incorporates a semi-automated annotation system utilizing trained object detection models. The software and framework are open-source.

Jumla Sign Language Annotation Tool: an overview

This paper describes an ongoing project on developing a new web-based tool for annotating Sign languages. This tool is used to annotate the First Qatari Sign Language dataset called Jumla Dataset: The Jumla Qatari Sign Language Corpus” with written Arabic text. The annotation of videos in Qatari Sign Language (QSL) takes input from signers to identify the Arabic glosses components toward representing the QSL in a written way with high accuracy, furthermore to the use of the annotation output in the development of computational Sign Language tools. The QSL annotation is based on an input of 4 videos recorded by deaf persons or Sign Language interpreters from different angles (front, left side, right side, and facial view). The output is a JSON file containing all the interpreted sentences given as an entry record. The glosses are annotated for each period and aligned with the Arabic content. Moreover, the presented tool, available as an open source, provides a management system to classify all records from cameras, motion capture systems, and edited files in addition to the possibility to create components for each gloss annotation terminology depending on the target Sign Language.

Jumla Sign Language Annotation Tool: an overview

This paper describes an ongoing project on developing a new web-based tool for annotating Sign languages. This tool is used to annotate the First Qatari Sign Language dataset called Jumla Dataset: The Jumla Qatari Sign Language Corpus” with written Arabic text. The annotation of videos in Qatari Sign Language (QSL) takes input from signers to identify the Arabic glosses components toward representing the QSL in a written way with high accuracy, furthermore to the use of the annotation output in the development of computational Sign Language tools. The QSL annotation is based on an input of 4 videos recorded by deaf persons or Sign Language interpreters from different angles (front, left side, right side, and facial view). The output is a JSON file containing all the interpreted sentences given as an entry record. The glosses are annotated for each period and aligned with the Arabic content. Moreover, the presented tool, available as an open source, provides a management system to classify all records from cameras, motion capture systems, and edited files in addition to the possibility to create components for each gloss annotation terminology depending on the target Sign Language.

A new artificial intelligence system successfully detectsand localises early neoplasia in Barrett's esophagus by using convolutional neural networks.

Background and aimsSeattle protocol biopsies for Barrett's Esophagus (BE) surveillance are labour intensive with low compliance. Dysplasia detection rates vary, leading to missed lesions. This can potentially be offset with computer aided detection. We have developed convolutional neural networks (CNNs) to identify areas of dysplasia and where to target biopsy.Methods119 Videos were collected in high‐definition white light and optical chromoendoscopy with i‐scan (Pentax Hoya, Japan) imaging in patients with dysplastic and non‐dysplastic BE (NDBE). We trained an indirectly supervised CNN to classify images as dysplastic/non‐dysplastic using whole video annotations to minimise selection bias and maximise accuracy. The CNN was trained using 148,936 video frames (31 dysplastic patients, 31 NDBE, two normal esophagus), validated on 25,161 images from 11 patient videos and tested on 264 iscan‐1 images from 28 dysplastic and 16 NDBE patients which included expert delineations. To localise targeted biopsies/delineations, a second directly supervised CNN was generated based on expert delineations of 94 dysplastic images from 30 patients. This was tested on 86 i‐scan one images from 28 dysplastic patients.FindingsThe indirectly supervised CNN achieved a per image sensitivity in the test set of 91%, specificity 79%, area under receiver operator curve of 93% to detect dysplasia. Per‐lesion sensitivity was 100%. Mean assessment speed was 48 frames per second (fps). 97% of targeted biopsy predictions matched expert and histological assessment at 56 fps. The artificial intelligence system performed better than six endoscopists.InterpretationOur CNNs classify and localise dysplastic Barrett's Esophagus potentially supporting endoscopists during surveillance.