Video Annotation Research Articles

Adoption of video annotation tool in enhancing students’ reflective ability level and communication competence

ABSTRACT Video-based education has shown a positive impact on teaching and learning. Hence, it has been increasingly adopted, and one form of video-based education is the use of video annotation tools (VATs). However, the adoption of VATs in life coaching courses has rarely been observed. The present investigation aims to examine the effect of VATs on enhancing students’ level of reflective thinking and communication competence during coaching practices in a sample of seventy-three undergraduates enrolled in the course ‘Life coaching with the aid of peer support’. During the coaching practices, two coaching sessions were video-taped, and the videos were uploaded on an online platform. Students were instructed to use VATs to give peer comments based on the performances of their peers. Their reflective thinking level and communication competence were also examined. In addition, four students participated in focus group interviews to explore the effectiveness of VATs. Two dependent t-tests demonstrated that both reflective thinking level and communication competence improved over time. Implications of the adoption of VATs in teaching and learning were discussed.

Enhancing teachers' noticing around mathematics teaching practices through video-based lesson study with peer coaching

PurposeThe purpose of this study is to examine what teachers notice in their own enactment of eight high leverage practices as well as the patterns of interactions between the teachers and their peers when participating in video-based lesson study.Design/methodology/approachEach teacher taught and uploaded video from one lesson to a platform, which allowed video annotation, for their lesson study team. There were nine lesson study teams. This study used a qualitative design to examine the teachers' comments on their own videos as well as the patterns in the comments between peers on lesson study teams.FindingsTeachers noticed both positive instantiations as well as opportunities for growth in their enactment of: using and connecting mathematical representations, posing purposeful questions and supporting students' productive struggle. Analysis displayed a pattern of exchanges where peers coached, validated, empathized and pushed each other beyond their comfort zone as critical peers.Research limitations/implicationsAlthough not all lesson study teams were made up of school-based teams and the teachers shared short recordings of their teaching, this research contributes to the understanding of how adapting lesson study by using video can help teachers notice their instantiation of teaching practices and peers can support and push one another towards ambitious instruction. Future research could extend this work by investigating the impact of video-based lesson study on teachers in isolated areas who may not have professional learning networks.Practical implicationsVideo-based LS may help to overcome barriers to the implementation of lesson study, such as the challenge of scheduling a common release time for lesson observation and the financial burden of funding substitute teachers for release time.Originality/valueThe current realities of COVID-19 creates an opportunity for mathematics educators to reimagine teacher professional development (PD) in ways that push the field forward. In light of this disruption, the authors propose an innovative model of utilizing video-based Lesson Study (LS; Lewis, 2002) with peer coaching to offer PD opportunities with methodological considerations for both mathematics researchers and teacher practitioners. The authors document and analyze a collection of online LSs that were taught by a focal teacher and recorded for the peers in the LS group. Video-based LS PD structure allowed the authors to examine how they can leverage this online model of LS to analyze student thinking and learn about teaching rich tasks in an online environment using eight teaching practices. Through their paper the authors will detail the necessary features of online LS specifically using a video annotation tool like Goreact and how video can be used to enhance the professional learning of the mathematics teaching practices (MTPs; NCTM, 2014) and the noticing of student thinking (Jacobs et al., 2010; Sherin and van Es, 2009; van Es and Sherin, 2002, 2008). In addition, the authors will document the norms that were established in the online LS community that impacted collaboration of LS teams and developed strong peer coaching relationships. The online LS PD design also supports collaboration of teachers from varying contexts, promotes professional growth and demonstrates how educators might leverage peer coaches as social capital within their schools to develop teachers along the professional continuum.

Automatic Annotation of American Football Video Footage for Game Strategy Analysis

Annotation and analysis of sports videos is a challenging task that, once accomplished, could provide various benefits to coaches, players, and spectators. In particular, American Football could benefit from such a system to provide assistance in statistics and game strategy analysis. Manual analysis of recorded American football game videos is a tedious and inefficient process. In this paper, as a first step to further our research for this unique application, we focus on locating and labeling individual football players from a single overhead image of a football play immediately before the play begins. A pre-trained deep learning network is used to detect and locate the players in the image. A ResNet is used to label the individual players based on their corresponding player position or formation. Our player detection and labeling algorithms obtain greater than 90% accuracy, especially for those skill positions on offense (Quarterback, Running Back, and Wide Receiver) and defense (Cornerback and Safety). Results from our preliminary studies on player detection, localization, and labeling prove the feasibility of building a complete American football strategy analysis system using artificial intelligence.

An assistive computer vision tool to automatically detect changes in fish behavior in response to ambient odor

The analysis of fish behavior in response to odor stimulation is a crucial component of the general study of cross-modal sensory integration in vertebrates. In zebrafish, the centrifugal pathway runs between the olfactory bulb and the neural retina, originating at the terminalis neuron in the olfactory bulb. Any changes in the ambient odor of a fish’s environment warrant a change in visual sensitivity and can trigger mating-like behavior in males due to increased GnRH signaling in the terminalis neuron. Behavioral experiments to study this phenomenon are commonly conducted in a controlled environment where a video of the fish is recorded over time before and after the application of chemicals to the water. Given the subtleties of behavioral change, trained biologists are currently required to annotate such videos as part of a study. This process of manually analyzing the videos is time-consuming, requires multiple experts to avoid human error/bias and cannot be easily crowdsourced on the Internet. Machine learning algorithms from computer vision, on the other hand, have proven to be effective for video annotation tasks because they are fast, accurate, and, if designed properly, can be less biased than humans. In this work, we propose to automate the entire process of analyzing videos of behavior changes in zebrafish by using tools from computer vision, relying on minimal expert supervision. The overall objective of this work is to create a generalized tool to predict animal behaviors from videos using state-of-the-art deep learning models, with the dual goal of advancing understanding in biology and engineering a more robust and powerful artificial information processing system for biologists.

‘I gave that cue.’ Integrating dance studies, praxeology, and computational perspectives to model change in the case study of William Forsythe’s Duo

ABSTRACT Choreographer William Forsythe calls the work Duo a ‘project’—reflecting the piece’s long history of vicissitudes from 1996 to the present. We attempt to visualize continuity and change over several iterations of Duo, spanning a period of 20 years. Our methods involved graphical and statistical approaches to performance video annotation, considering seven videos acquired from Forsythe’s private archive. Collaboration with Duo dancers was critical to develop this choreographic knowledge. The duet Duo was chosen to focus on annotation of partnering, choreographic structure, and interpretation; the case study furthermore enabled review of annotation methods from Forsythe’s Synchronous Objects for One Flat Thing, reproduced (Forsythe et al. 2009) and built upon prior research of entrainment in Duo (Waterhouse, Watts, and Bläsing 2014). Studying a choreography longitudinally, with close regard of the performers’ testimonies and digital traces, the problem required innovative methods. For this article, we focus on how annotation was used within this project. We outline our particular interdisciplinary approach, merging perspectives from dance studies, praxeology and creative coding. We present the language and concepts of annotation chosen, technical tools used for annotating, procedures of annotation analysis, and conclusions of the research. Thereby we present novel visualizations of choreographic process.

Differentiating (an)notation practices: an artist-scholar’s observation

ABSTRACT Video annotation is an emergent practice and not (yet) a common method in dance studies or research. Subsequently, there are limited accounts that detail the practice of using annotation in dance but those that are available point to how annotation serves diverse and particular purposes. However, a common understanding of what annotation is does not theoretically cohere. Furthermore, the tendency to use the terms annotation and notation synonymously conflates these practices and risks overlooking the significant contributions of each. In discussing my experience, reflections, and observations of working with four different approaches to annotation I offer an understanding of what it offers in analysing and transmitting ideas about dance from an artist-scholar's perspective. Crucially, drawing from Bernard Stiegler's philosophy of technology, I position annotations as technical memories created in dialogue with existing mnemotechnical forms, or technical objects. Such characterisation illuminates how annotation helps to overcome limitations of documentary forms and highlight information otherwise missing or previously unnoticed. To further emphasise annotation as a method of amplification I compare my experience of annotation and of Labanotation to highlight the similarities and differences between these distinctive methodological tools. While the examples primarily focus on dance the insight developed in this article is valuable for other fields working with time-based media.

Comparative video annotation and visual literacy: performance analysis of Rina Yerushalmi’s theatre language

ABSTRACT The growing availability of video recordings of theatre performances, a phenomenon that has increased during Covid19 as theatres worldwide share videos online, affects the field of theatre and performance studies. Video recordings of theatre performances are archival documents and mediated ‘performance texts’ that enable new kinds of performance analysis and studying bodily practices. This paper addresses annotative methodologies as part of the research project, ‘The Art of Adaptation: The Theatre of Rina Yerushalmi and the Itim Ensemble.’ The project studies the video archive of the ensemble, which includes recordings of full productions and rehearsal processes. We discuss three kinds of digital comparative annotative methodologies to show how annotation can be used as a research tool for performance analysis: (1) Accumulative annotation: analysis of multi-layered theatrical sequences, revealing theatrical moments in their multiplicity. (2) Annotation of different scenes in a corpus of works: juxtaposing scenes from different productions enables the articulation of repetitive performative patterns, embodied practices, and visual images that construct a theatre language. (3) Annotation of the same scene at different phases: juxtaposing video recordings of the same scene at different moments of its development reveals nuanced sets of information about directorial choices, acting, movement, duration, and more.

Robotic Resection of a Nerve Sheath Tumor Via a Retroperitoneal Approach.

Resection of large nerve sheath tumors in the lumbar spine using minimally invasive approaches is challenging, as approaches to tumors in this region may require facetectomy or partial resection of adjacent ribs for access to the involved neuroforamen and instrumentation across the involved joint to prevent subsequent kyphotic deformity. To describe a robot-assisted retroperitoneal approach for resection of a lumbar nerve sheath tumor, obviating the need for facetectomy and instrumentation. The operation is described, together with intraoperative images and an annotated video, in the context of a schwannoma arising from the right L1 root. The operation was performed by a urologic surgeon and a neurosurgeon. The patient was placed in lateral position, and the da Vinci Xi robot was used for retroperitoneal access via 5 ports along the right flank. Ultrasound was used to localize the tumor within the psoas. The tumor capsule was defined and released. Encountered nerves were stimulated, allowing small sensory nerves to be identified and safely divided. The tumor was traced into the right L1-L2 neuroforamen and removed. Complete en bloc resection of the tumor was achieved, including the paraspinal and foraminal components, without any removal of bone and without violation of the dura. In selected patients, a robot-assisted retroperitoneal approach represents a minimally invasive alternative to traditional approaches for resection of lumbar nerve sheath tumors. This approach obviates the need for bone removal and instrumented spinal fusion. Interdisciplinary collaboration, as well as use of adjunctive technologies, including intraoperative ultrasound and neurophysiologic monitoring, is advised.

Challenges in surgical video annotation

Annotation of surgical video is important for establishing ground truth in surgical data science endeavors that involve computer vision. With the growth of the field over the last decade, several challenges have been identified in annotating spatial, temporal, and clinical elements of surgical video as well as challenges in selecting annotators. In reviewing current challenges, we provide suggestions on opportunities for improvement and possible next steps to enable translation of surgical data science efforts in surgical video analysis to clinical research and practice.

Development and Verification of a System to Support Cooperative Learning Based on Peer Review by Visual Video Annotation on Portable Mobile Device

Development and Verification of a System to Support Cooperative Learning Based on Peer Review by Visual Video Annotation on Portable Mobile Device

Smart hand gestures recognition using K-NN based algorithm for video annotation purposes

<span>Sign languages are the most basic and natural form of languages which were used even before the evolution of spoken languages. These sign languages were developed using various sign "gestures" that are made using hand palm. Such gestures are called "hand gestures". Hand gestures are being widely used as an international assistive communication method for deaf people and many life aspects such as sports, traffic control and religious acts. However, the meanings of hand gestures vary among different civilization cultures. Therefore, because of the importance of understanding the meanings of hand gestures, this study presents a procedure whichcan translate such gestures into an annotated explanation. The proposed system implements image and video processing which are recently conceived as one of the most important technologies. The system initially, analyzes a classroom video as an input, and then extracts the vocabulary of twenty gestures. Various methods have been applied sequentially, namely: motion detection, RGB to HSV conversion, and noise removing using labeling algorithms. The extraction of hand parameters is determined by a K-NN algorithm to eventually determine the hand gesture and, hence showing their meanings. To estimate the performance of the proposed method, an experiment using a hand gesture database is performed. The results showed that the suggested method has an average recognition rate of 97%. </span>

Synthetic Expressions are Better Than Real for Learning to Detect Facial Actions.

Critical obstacles in training classifiers to detect facial actions are the limited sizes of annotated video databases and the relatively low frequencies of occurrence of many actions. To address these problems, we propose an approach that makes use of facial expression generation. Our approach reconstructs the 3D shape of the face from each video frame, aligns the 3D mesh to a canonical view, and then trains a GAN-based network to synthesize novel images with facial action units of interest. To evaluate this approach, a deep neural network was trained on two separate datasets: One network was trained on video of synthesized facial expressions generated from FERA17; the other network was trained on unaltered video from the same database. Both networks used the same train and validation partitions and were tested on the test partition of actual video from FERA17. The network trained on synthesized facial expressions outperformed the one trained on actual facial expressions and surpassed current state-of-the-art approaches.

Asynchronous, digital teaching in times of COVID-19: a teaching example from general practice.

Background: The SARS-CoV-2 pandemic had a strong impact on academic teaching and could change it sustainably. Ad hoc digitization of teaching had to be carried out. General practice teaching situation: Education in general practice at the Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU) offers, in addition to the main lecture, various elective courses, clinical traineeships, internship as well as the elective part in the final practical year. The main lecture and one clinical elective course were offered digitally in the summer term 2020.Digital methods: In the main lecture, an adapted inverted-classroom concept was used. Podcasts and audio annotated videos were provided. Teaching materials were reflected via a weekly, 1hr video consultation and in a forum. An asynchronous learning module was developed for the elective course “Smart Decision-making in Clinical Practice”. Each module consisted of course preparation, podcasts and follow-ups as well as a supervised forum.Results: The main lecture (response rate n=115/170; 67.6%) was rated “very good” on average. The same applies to the commented videos. The forum, reflective video consultation and teaching materials were rated “good” on average. The predominantly desired forms of presence were “Focus on virtual with in-depth presence phases” (n=54) and “Focus on presence phases, virtual support only” (n=37).Discussion and implications: The digital restructuring enables students to work on the course contents independently. This requires self-regulation strategies, which in future shall be taught through portfolio work. The teaching focus shifts from a passive teaching format to an interactive one. First evaluation results showed a very good acceptance by the students.

Deep learning for surgical phase recognition using endoscopic videos.

Operating room planning is a complex task as pre-operative estimations of procedure duration have a limited accuracy. This is due to large variations in the course of procedures. Therefore, information about the progress of procedures is essential to adapt the daily operating room schedule accordingly. This information should ideally be objective, automatically retrievable and in real-time. Recordings made during endoscopic surgeries are a potential source of progress information. A trained observer is able to recognize the ongoing surgical phase from watching these videos. The introduction of deep learning techniques brought up opportunities to automatically retrieve information from surgical videos. The aim of this study was to apply state-of-the art deep learning techniques on a new set of endoscopic videos to automatically recognize the progress of a procedure, and to assess the feasibility of the approach in terms of performance, scalability and practical considerations. A dataset of 33 laparoscopic cholecystectomies (LC) and 35 total laparoscopic hysterectomies (TLH) was used. The surgical tools that were used and the ongoing surgical phases were annotated in the recordings. Neural networks were trained on a subset of annotated videos. The automatic recognition of surgical tools and phases was then assessed on another subset. The scalability of the networks was tested and practical considerations were kept up. The performance of the surgical tools and phase recognition reached an average precision and recall between 0.77 and 0.89. The scalability tests showed diverging results. Legal considerations had to be taken into account and a considerable amount of time was needed to annotate the datasets. This study shows the potential of deep learning to automatically recognize information contained in surgical videos. This study also provides insights in the applicability of such a technique to support operating room planning.

The detection, tracking, and temporal action localisation of swimmers for automated analysis

It is very important for swimming coaches to analyse a swimmer’s performance at the end of each race, since the analysis can then be used to change strategies for the next round. Coaches rely heavily on statistics, such as stroke length and instantaneous velocity, when analysing performance. These statistics are usually derived from time-consuming manual video annotations. To automatically obtain the required statistics from swimming videos, we need to solve the following four challenging computer vision tasks: swimmer head detection; tracking; stroke detection; and camera calibration. We collectively solve these problems using a two-phased deep learning approach, we call Deep Detector for Actions and Swimmer Heads (DeepDASH). DeepDASH achieves a 20.8% higher F1 score for swimmer head detection and operates 6 times faster than the popular Faster R-CNN object detector. We also propose a hierarchical tracking algorithm based on the existing SORT algorithm which we call HISORT. HISORT produces significantly longer tracks than SORT by preserving swimmer identities for longer periods of time. Finally, DeepDASH achieves an overall F1 score of 97.5% for stroke detection across all four swimming stroke styles.

Artificial Intelligence-Enabled Traffic Monitoring System

Manual traffic surveillance can be a daunting task as Traffic Management Centers operate a myriad of cameras installed over a network. Injecting some level of automation could help lighten the workload of human operators performing manual surveillance and facilitate making proactive decisions which would reduce the impact of incidents and recurring congestion on roadways. This article presents a novel approach to automatically monitor real time traffic footage using deep convolutional neural networks and a stand-alone graphical user interface. The authors describe the results of research received in the process of developing models that serve as an integrated framework for an artificial intelligence enabled traffic monitoring system. The proposed system deploys several state-of-the-art deep learning algorithms to automate different traffic monitoring needs. Taking advantage of a large database of annotated video surveillance data, deep learning-based models are trained to detect queues, track stationary vehicles, and tabulate vehicle counts. A pixel-level segmentation approach is applied to detect traffic queues and predict severity. Real-time object detection algorithms coupled with different tracking systems are deployed to automatically detect stranded vehicles as well as perform vehicular counts. At each stage of development, interesting experimental results are presented to demonstrate the effectiveness of the proposed system. Overall, the results demonstrate that the proposed framework performs satisfactorily under varied conditions without being immensely impacted by environmental hazards such as blurry camera views, low illumination, rain, or snow.

Efficient method for evaluation and hardware-in-loop testing of electro–optical tracking system

Object tracking technology is being continually improved to ensure robust tracking performance under real conditions, such as cluttered background, fast motion, and lengthy occlusions. The performance of these video trackers depends significantly on evaluation methodologies adopted during the assessment of tracking algorithms. Most of these video trackers are evaluated using video sequences annotated with the simulated visual attributes of objects. Generating such sequences with the requisite attributes is laborious and time consuming. These annotated video sequences may fail to address real challenges; hence, the trackers must be tested using available opportunity targets in lengthy field trials. We present an innovative scheme for the hardware-in-loop testing of such video trackers for efficient evaluation under complex background scenarios. The designed simulator can provide the real-time visual characteristics of object parameters with a live background without halting the evaluation process. The proposed method facilitates the efficient evaluation of a closed-loop video tracker by providing a virtually endless video sequence containing realistic visual attributes of potential objects. In addition, the validation of the tracker’s performance in realistic scenarios, the comparative performance of tracking algorithms under varying operational conditions, and an in-depth analysis of the tracker system parameters are presented.

Real-Life Gait Performance as a Digital Biomarker for Motor Fluctuations: The Parkinson@Home Validation Study.

BackgroundWearable sensors have been used successfully to characterize bradykinetic gait in patients with Parkinson disease (PD), but most studies to date have been conducted in highly controlled laboratory environments.ObjectiveThis paper aims to assess whether sensor-based analysis of real-life gait can be used to objectively and remotely monitor motor fluctuations in PD.MethodsThe Parkinson@Home validation study provides a new reference data set for the development of digital biomarkers to monitor persons with PD in daily life. Specifically, a group of 25 patients with PD with motor fluctuations and 25 age-matched controls performed unscripted daily activities in and around their homes for at least one hour while being recorded on video. Patients with PD did this twice: once after overnight withdrawal of dopaminergic medication and again 1 hour after medication intake. Participants wore sensors on both wrists and ankles, on the lower back, and in the front pants pocket, capturing movement and contextual data. Gait segments of 25 seconds were extracted from accelerometer signals based on manual video annotations. The power spectral density of each segment and device was estimated using Welch’s method, from which the total power in the 0.5- to 10-Hz band, width of the dominant frequency, and cadence were derived. The ability to discriminate between before and after medication intake and between patients with PD and controls was evaluated using leave-one-subject-out nested cross-validation.ResultsFrom 18 patients with PD (11 men; median age 65 years) and 24 controls (13 men; median age 68 years), ≥10 gait segments were available. Using logistic LASSO (least absolute shrinkage and selection operator) regression, we classified whether the unscripted gait segments occurred before or after medication intake, with mean area under the receiver operator curves (AUCs) varying between 0.70 (ankle of least affected side, 95% CI 0.60-0.81) and 0.82 (ankle of most affected side, 95% CI 0.72-0.92) across sensor locations. Combining all sensor locations did not significantly improve classification (AUC 0.84, 95% CI 0.75-0.93). Of all signal properties, the total power in the 0.5- to 10-Hz band was most responsive to dopaminergic medication. Discriminating between patients with PD and controls was generally more difficult (AUC of all sensor locations combined: 0.76, 95% CI 0.62-0.90). The video recordings revealed that the positioning of the hands during real-life gait had a substantial impact on the power spectral density of both the wrist and pants pocket sensor.ConclusionsWe present a new video-referenced data set that includes unscripted activities in and around the participants’ homes. Using this data set, we show the feasibility of using sensor-based analysis of real-life gait to monitor motor fluctuations with a single sensor location. Future work may assess the value of contextual sensors to control for real-world confounders.

CNN-based single object detection and tracking in videos and its application to drone detection

This paper presents convolutional neural network (CNN)-based single object detection and tracking algorithms. CNN-based object detection methods are directly applicable to static images, but not to videos. On the other hand, model-free visual object tracking methods cannot detect an object until a ground truth bounding box of the target is provided. Moreover, many annotated video datasets of the target object are required to train both the object detectors and visual trackers. In this work, three simple yet effective object detection and tracking algorithms for videos are proposed to efficiently combine a state-of-the-art object detector and visual tracker for circumstances in which only a few static images of the target are available for training. The proposed algorithms are tested using a drone detection task and the experimental results demonstrated their effectiveness.

Self-Supervised Deep TripleNet for Video Object Segmentation

Most of previous video object segmentation methods require a large amount of pixel-level annotated video data to construct a robust model. It is quite expensive to label segmentation mask in video. In this article, we propose a self-supervised triplenet for video object segmentation, which only leverages nearly unlimited unlabeled video data in training phase. Our method consists of two modules, i.e., the temporal motion module and the appearance matching module. The temporal motion module is trained based on the pixel correspondence between two video frames in a self-supervised manner, which models the motion patterns between two video frames and propagates the labels from one frame to another. Meanwhile, the appearance matching module encodes the reference frame and its corresponding mask to generate the segmentation mask of the same object in target frame. The appearance matching module can adjust and refine the output results of temporal motion module, and avoid error accumulation by matching the reference appearance. In order to train the appearance matching module in self-supervised manner, we propose two mask generation strategies: foreground region mask generation and random color region mask generation. Extensive experiments conducted on four challenging video object segmentation datasets, i.e., DAVIS-2017, Youtube-VOS, DAVIS-2016 and SegTrack v2, demonstrate that the proposed method performs favorable against the state-of-the-art self-supervised methods, and performs even competitively with fully-supervised methods. We also show our self-supervised approach has actually superior generalizability to the majority of supervised methods.