Video Labeling Research Articles

Detection and Segmentation of Mouth Region in Stereo Stream Using YOLOv6 and DeepLab v3+ Models for Computer-Aided Speech Diagnosis in Children

This paper describes a multistage framework for face image analysis in computer-aided speech diagnosis and therapy. Multimodal data processing frameworks have become a significant factor in supporting speech disorders’ treatment. Synchronous and asynchronous remote speech therapy approaches can use audio and video analysis of articulation to deliver robust indicators of disordered speech. Accurate segmentation of articulators in video frames is a vital step in this agenda. We use a dedicated data acquisition system to capture the stereovision stream during speech therapy examination in children. Our goal is to detect and accurately segment four objects in the mouth area (lips, teeth, tongue, and whole mouth) during relaxed speech and speech therapy exercises. Our database contains 17,913 frames from 76 preschool children. We apply a sequence of procedures employing artificial intelligence. For detection, we train the YOLOv6 (you only look once) model to catch each of the three objects under consideration. Then, we prepare the DeepLab v3+ segmentation model in a semi-supervised training mode. As preparation of reliable expert annotations is exhausting in video labeling, we first train the network using weak labels produced by initial segmentation based on the distance-regularized level set evolution over fuzzified images. Next, we fine-tune the model using a portion of manual ground-truth delineations. Each stage is thoroughly assessed using the independent test subset. The lips are detected almost perfectly (average precision and F1 score of 0.999), whereas the segmentation Dice index exceeds 0.83 in each articulator, with a top result of 0.95 in the whole mouth.

Auxiliary audio–textual modalities for better action recognition on vision-specific annotated videos

Most current audio–visual datasets are class-relevant, where audio and visual modalities are annotated. Thus, current audio–visual recognition methods apply cross-modality attention or modality fusion. However, leveraging the audio modality effectively in vision-specific videos for human activity recognition is of particular challenge. We address this challenge by proposing a novel audio–visual recognition framework that effectively leverages audio modality in any vision-specific annotated dataset. The proposed framework employs language models (e.g., GPT-3, CPT-text, BERT) for building a semantic audio–video label dictionary (SAVLD) that serves as a bridge between audio and video datasets by mapping each video label to its most K-relevant audio labels. Then, SAVLD along with a pre-trained audio multi-label model are used to estimate the audio–visual modality relevance. Accordingly, we propose a novel learnable irrelevant modality dropout (IMD) to completely drop the irrelevant audio modality and fuse only the relevant modalities. Finally, for the efficiency of the proposed multimodal framework, we present an efficient two-stream video Transformer to process the visual modalities (i.e., RGB frames and optical flow). The final predictions are re-ranked with GPT-3 recommendations of the human activity classes. GPT-3 provides high-level recommendations using the labels of the detected visual objects and the audio predictions of the input video. Our framework demonstrated a remarkable performance on the vision-specific annotated datasets Kinetics400 and UCF-101 by outperforming most relevant human activity recognition methods.

Using a Human-Centered Design Process to Evaluate and Optimize User Experience of a Website (InPACT at Home) to Promote Youth Physical Activity: Case Study.

Web-based physical activity interventions often fail to reach the anticipated public health impact due to insufficient use by the intended audiences. The purpose of this study was to use a human-centered design process to optimize the user experience of the Interrupting Prolonged sitting with ACTivity (InPACT) at Home website to promote youth physical activity participation. Qualitative interviews were conducted to assess engagement and pain points with the InPACT at Home website. Interview data were used to create affinity maps to identify themes of user responses, conduct a heuristic evaluation according to Nielsen's usability heuristics framework, and complete a competitive analysis to identify the strengths and weaknesses of competitors who offered similar products. Key themes from end user interviews included liking the website design, finding the website difficult to navigate, and wanting additional features (eg, library of watched videos). The website usability issues identified were lack of labeling and categorization of exercise videos, hidden necessary actions and options hindering users from decision-making, error-prone conditions, and high cognitive load of the website. Competitive analysis results revealed that YouTube received the highest usability ratings followed by the Just Dance and Presidential Youth Fitness Program websites. Human-centered design approaches are useful for bringing end users and developers together to optimize user experience and impact public health. Future research is needed to examine the effectiveness of the InPACT at Home website redesign to attract new users and retain current users, with the end goal of increasing youth physical activity engagement.

A Video Action Recognition Method via Dual-Stream Feature Fusion Neural Network with Attention

Video action recognition is a technique for automatically determining the category of a video action. It is necessary to design an efficient video action recognition algorithm to predict video labels. This work proposes a video action recognition model based on dual-stream information fusion with attention mechanisms (DSIFAM), which consists of three different sub-modules. First, this proposes an improved keyframe extraction method (IKFE). Based on K-means clustering, this uses convolutional features to calculate the similarity between video frames instead of pixel points. After obtaining preliminary clustering results, the method performs secondary optimization to obtain more representative keyframes. Second, this proposes a video action recognition model based on dual-stream information fusion (DSIF). The method introduces ConvLSTM in the spatial stream and uses P3D instead of the original convolutional network in the temporal stream, which can better extract spatial-temporal information and improve the classification performance. Third, this designs a multi-scale attention mechanism (MSAM) to enhance the feature extraction stage and obtain higher quality classification features. The resulting features are more prominent and have stronger representation capabilities. Finally, this work conducts systematic experiments on different datasets, the results verify the superiority of DSIFAM for video action recognition.

Artificial intelligence and robotic surgical education

There are numerous barriers in robotic surgical training, including reliance on observational learning, low-quality feedback, and inconsistent assessment. Artificial intelligence (AI) offers potential solutions to these central problems in robotic surgical education and may allow for more efficient and efficacious training. Three key areas in which AI has particular relevance to robotic surgical education are video labeling, feedback, and assessment. Video labeling refers to the automated designation of prespecified categories to operative videos. Numerous prior studies have applied AI for video labeling, particularly for retrospective educational review after an operation. Video labeling allows learners and their instructors to rapidly identify critical parts of an operative video. We recommend incorporating AI-based video labeling into robotic surgical education where available. AI also offers a mechanism by which reliable feedback can be provided in robotic surgery. Feedback through AI harnesses automated performance metrics (APMs) and natural language processing (NLP) to provide actionable and descriptive plans to learners while reducing faculty assessment burden. We recommend combining supervised AI-generated, APM-based feedback with expert-based feedback to allow surgeons and trainees to reflect on metrics like bimanual dexterity and efficiency. Finally, summative assessment by AI could allow for automated appraisal of surgeons or surgical trainees. However, AI-based assessment remains limited by concerns around bias and opaque processes. Several studies have applied computer vision to compare AI-based assessment with expert-completed rating scales, though such work remains investigational. At this time, we recommend against the use of AI for summative assessment pending additional validity evidence. Overall, AI offers solutions and promising future directions by which to address multiple educational challenges in robotic surgery. Through advances in video labeling, feedback, and assessment, AI has demonstrated ways by which to increase the efficiency and efficacy of robotic surgical education.

Weakly Supervised Tracklet Association Learning With Video Labels for Person Re-Identification.

Supervised person re-identification (re-id) methods require expensive manual labeling costs. Although unsupervised re-id methods can reduce the requirement of the labeled datasets, the performance of these methods is lower than the supervised alternatives. Recently, some weakly supervised learning-based person re-id methods have been proposed, which is a balance between supervised and unsupervised learning. Nevertheless, most of these models require another auxiliary fully supervised datasets or ignore the interference of noisy tracklets. To address this problem, in this work, we formulate a weakly supervised tracklet association learning (WS-TAL) model only leveraging the video labels. Specifically, we first propose an intra-bag tracklet discrimination learning (ITDL) term. It can capture the associations between person identities and images by assigning pseudo labels to each person image in a bag. And then, the discriminative feature for each person is learned by utilizing the obtained associations after filtering the noisy tracklets. Based on that, a cross-bag tracklet association learning (CTAL) term is presented to explore the potential tracklet associations between bags by mining reliable positive tracklet pairs and hard negative pairs. Finally, these two complementary terms are jointly optimized to train our re-id model. Extensive experiments on the weakly labeled datasets demonstrate that WS-TAL achieves 88.1% and 90.3% rank-1 accuracy on the MARS and DukeMTMC-VideoReID datasets respectively. The performance of our model surpasses the state-of-the-art weakly supervised models by a large margin, even outperforms some fully supervised re-id models.

On-the-fly point annotation for fast medical video labeling.

In medical research, deep learning models rely on high-quality annotated data, a process often laborious and time-consuming. This is particularly true for detection tasks where bounding box annotations are required. The need to adjust two corners makes the process inherently frame-by-frame. Given the scarcity of experts' time, efficient annotation methods suitable for clinicians are needed. We propose an on-the-fly method for live video annotation to enhance the annotation efficiency. In this approach, a continuous single-point annotation is maintained by keeping the cursor on the object in a live video, mitigating the need for tedious pausing and repetitive navigation inherent in traditional annotation methods. This novel annotation paradigm inherits the point annotation's ability to generate pseudo-labels using a point-to-box teacher model. We empirically evaluate this approach by developing a dataset and comparing on-the-fly annotation time against traditional annotation method. Using our method, annotation speed was faster than the traditional annotation technique. We achieved a mean improvement of AP@50 over conventional method at equivalent annotation budgets on the developed dataset. Without bells and whistles, our approach offers a significant speed-up in annotation tasks. It can be easily implemented on any annotation platform to accelerate the integration of deep learning in video-based medical research.

PADELVIC: Multicamera videos and motion capture data of an amateur padel match

Recent advances in computer vision and deep learning techniques have opened new possibilities regarding the automatic labeling of sport videos. However, an essential requirement for supervised techniques is the availability of accurately labeled training datasets. In this paper we present PadelVic, an annotated dataset of an amateur padel match which consists of multi-view video streams, accurate motion capture data of one of the players, as well as synthetic videos specifically designed to serve as training sets for convolutional neural networks estimating positional data from videos. As a demonstration of one of the applications of the dataset, we present a system for the accurate prediction of the center-of-mass of the players projected onto the court plane, from a single-view video of the match.

Investigating the Influence of Convolutional Operations on LSTM Networks in Video Classification

Video classification holds a foundational position in the realm of computer vision, involving the categorization and labeling of videos based on their content. Its significance resonates across various applications, including video surveil-lance, content recommendation, action recognition, video indexing, and more. The primary objective of video classification is to automatically analyze and comprehend the visual information embedded in videos, facilitating the efficient organization, retrieval, and interpretation of extensive video collections. The integration of convolutional neural networks (CNNs) and long short-term memory (LSTM) networks has brought about a revolution in video classification. This fusion effectively captures both spatial and temporal dependencies within video sequences, leveraging the strengths of CNNs in extracting spatial features and LSTMs in modeling sequential and temporal information. ConvLSTM and LRCN (Long-term Recurrent Convolutional Networks) are two widely embraced architectures that embody this fusion. This paper seeks to investigate the impact of convolutions on LSTM networks in the context of video classification, aiming to compare the performance of ConvLSTM and LRCN.

Improvement of continuous emotion recognition of temporal convolutional networks with incomplete labels

AbstractVideo‐based emotion recognition has been a long‐standing research topic for computer scientists and psychiatrists. In contrast to traditional discrete emotional models, emotion recognition based on continuous emotional models can better describe the progression of emotions. Quantitative analysis of emotions will have crucial impacts on promoting the development of intelligent products. The current solutions to continuous emotion recognition still have many issues. The original continuous emotion dataset contains incomplete data annotations, and the existing methods often ignore temporal information between frames. The following measures are taken in response to the above problems. Initially, aiming at the problem of incomplete video labels, the correlation between discrete and continuous video emotion labels is used to complete the dataset labels. This correlation is used to propose a mathematical model to fill the missing labels of the original dataset without adding data. Moreover, this paper proposes a continuous emotion recognition network based on an optimized temporal convolutional network, which adds a feature extraction submodule and a residual module to retain shallow features while improving the feature extraction ability. Finally, validation experiments on the Aff‐wild2 dataset achieved accuracies of 0.5159 and 0.65611 on the valence and arousal dimensions, respectively, by adopting the above measures.

Automatic labeling of Parkinson's Disease gait videos with weak supervision.

Motor dysfunction in Parkinson's Disease (PD) patients is typically assessed by clinicians employing the Movement Disorder Society's Unified Parkinson's Disease Rating Scale (MDS-UPDRS). Such comprehensive clinical assessments are time-consuming, expensive, semi-subjective, and may potentially result in conflicting labels across different raters. To address this problem, we propose an automatic, objective, and weakly-supervised method for labeling PD patients' gait videos. The proposed method accepts videos of patients and classifies their gait scores as normal (Gait score in MDS-UPDRS = 0) or PD (MDS-UPDRS ≥ 1). Unlike previous work, the proposed method does not require a priori MDS-UPDRS ratings for training, utilizing only domain-specific knowledge obtained from neurologists. We propose several labeling functions that classify patients' gait and use a generative model to learn the accuracy of each labeling function in a self-supervised manner. Since results depended upon the estimated values of the patients' 3D poses, and existing pre-trained 3D pose estimators did not yield accurate results, we propose a weakly-supervised 3D human pose estimation method for fine-tuning pre-trained models in a clinical setting. Using leave-one-out evaluations, the proposed method obtains an accuracy of 89% on a dataset of 29 PD subjects - a significant improvement compared to previous work by 7%-10% depending upon the dataset. The method obtained state-of-the-art results on the Human3.6M dataset. Our results suggest that the use of labeling functions may provide a robust means to interpret and classify patient-oriented videos involving motor tasks.

Transferability of a Sensing Mattress for Posture Classification from Research into Clinics.

Sleep is crucial in rehabilitation processes, promoting neural plasticity and immune functions. Nocturnal body postures can indicate sleep quality and frequent repositioning is required to prevent bedsores for bedridden patients after a stroke or spinal cord injury. Polysomnography (PSG) is considered the gold standard for sleep assessment. Unobtrusive methods for classifying sleep body postures have been presented with similar accuracy to PSG, but most evaluations have been done in research lab environments. To investigate the challenges in the usability of a previously validated device in a clinical setting, we recorded the sleep posture of 17 patients with a sensorized mattress. Ground-truth labels were collected automatically from a PSG device. In addition, we manually labeled the body postures using video data. This allowed us also to evaluate the quality of the PSG labels. We trained neural networks based on the VGG-3 architecture to classify lying postures and used a self-label correction method to account for noisy labels in the training data. The models trained with the video labels achieved a higher classification accuracy than those trained with the PSG labels (0.79 vs. 0.68). The self-label correction could further increase the models' scores based on video and PSG labels to 0.80 and 0.70, respectively. Unobtrusive sensors validated in clinics can, therefore, potentially improve the quality of care for bedridden patients and advance the field of rehabilitation.

Real-World Anomaly Detection in Video Using Spatio-Temporal Features Analysis for Weakly Labelled Data with Auto Label Generation

Detecting anomalies in videos is a complex task due to diverse content, noisy labeling, and a lack of frame-level labeling. To address these challenges in weakly labeled datasets, we propose a novel custom loss function in conjunction with the multi-instance learning (MIL) algorithm. Our approach utilizes the UCF Crime and ShanghaiTech datasets for anomaly detection. The UCF Crime dataset includes labeled videos depicting a range of incidents such as explosions, assaults, and burglaries, while the ShanghaiTech dataset is one of the largest anomaly datasets, with over 400 video clips featuring three different scenes and 130 abnormal events. We generated pseudo labels for videos using the MIL technique to detect frame-level anomalies from video-level annotations, and to train the network to distinguish between normal and abnormal classes. We conducted extensive experiments on the UCF Crime dataset using C3D and I3D features to test our model's performance. For the ShanghaiTech dataset, we used I3D features for training and testing. Our results show that with I3D features, we achieve an 84.6% frame-level AUC score for the UCF Crime dataset and a 92.27% frame-level AUC score for the ShanghaiTech dataset, which are comparable to other methods used for similar datasets.

A Review of Medical Diagnostic Video Analysis Using Deep Learning Techniques

The automated analysis of medical diagnostic videos, such as ultrasound and endoscopy, provides significant benefits in clinical practice by improving the efficiency and accuracy of diagnosis. Deep learning techniques show remarkable success in analyzing these videos by automating tasks such as classification, detection, and segmentation. In this paper, we review the application of deep learning techniques for analyzing medical diagnostic videos, with a focus on ultrasound and endoscopy. The methodology for selecting the papers consists of two major steps. First, we selected around 350 papers based on the relevance of their titles to our topic. Second, we chose the research articles that focus on deep learning and medical diagnostic videos based on our inclusion and exclusion criteria. We found that convolutional neural networks (CNNs) and long short-term memory (LSTM) are the two most commonly used models that achieve good results in analyzing different types of medical videos. We also found various limitations and open challenges. We highlight the limitations and open challenges in this field, such as labeling and preprocessing of medical videos, class imbalance, and time complexity, as well as incorporating expert knowledge, k-shot learning, live feedback from experts, and medical history with video data. Our review can encourage collaborative research with domain experts and patients to improve the diagnosis of diseases from medical videos.

Concurrent validity of machine learning-classified functional upper extremity use from accelerometry in chronic stroke.

Objective: This study aims to investigate the validity of machine learning-derived amount of real-world functional upper extremity (UE) use in individuals with stroke. We hypothesized that machine learning classification of wrist-worn accelerometry will be as accurate as frame-by-frame video labeling (ground truth). A second objective was to validate the machine learning classification against measures of impairment, function, dexterity, and self-reported UE use. Design: Cross-sectional and convenience sampling. Setting: Outpatient rehabilitation. Participants: Individuals (>18years) with neuroimaging-confirmed ischemic or hemorrhagic stroke >6-months prior (n = 31) with persistent impairment of the hemiparetic arm and upper extremity Fugl-Meyer (UEFM) score = 12-57. Methods: Participants wore an accelerometer on each arm and were video recorded while completing an "activity script" comprising activities and instrumental activities of daily living in a simulated apartment in outpatient rehabilitation. The video was annotated to determine the ground-truth amount of functional UE use. Main outcome measures: The amount of real-world UE use was estimated using a random forest classifier trained on the accelerometry data. UE motor function was measured with the Action Research Arm Test (ARAT), UEFM, and nine-hole peg test (9HPT). The amount of real-world UE use was measured using the Motor Activity Log (MAL). Results: The machine learning estimated use ratio was significantly correlated with the use ratio derived from video annotation, ARAT, UEFM, 9HPT, and to a lesser extent, MAL. Bland-Altman plots showed excellent agreement between use ratios calculated from video-annotated and machine-learning classification. Factor analysis showed that machine learning use ratios capture the same construct as ARAT, UEFM, 9HPT, and MAL and explain 83% of the variance in UE motor performance. Conclusion: Our machine learning approach provides a valid measure of functional UE use. The accuracy, validity, and small footprint of this machine learning approach makes it feasible for measurement of UE recovery in stroke rehabilitation trials.

Video labelling robot-assisted radical prostatectomy and the role of artificial intelligence (AI): training a novice

Video labelling is the assigning of meaningful information to raw videos. With the evolution of artificial intelligence and its intended incorporation into the operating room, video datasets can be invaluable tools for education and the training of intelligent surgical workflow systems through computer vision. However, the process of manual labelling of video datasets can prove costly and time-consuming for already busy practising surgeons. Twenty-five robot-assisted radical prostatectomy (RARP) procedures were recorded on Proximie, an augmented reality platform, anonymised and access given to a novice, who was trained to develop the knowledge and skills needed to accurately segment a full-length RARP procedure on a video labelling platform. A labelled video was subsequently randomly selected for assessment of accuracy by four practising urologists. Of the 25 videos allocated, 17 were deemed suitable for labelling, and 8 were excluded on the basis of procedure length and video quality. The labelled video selected for assessment was graded for accuracy of temporal labelling, with an average score of 93.1%, and a range of 85.6–100%. The self-training of a novice in the accurate segmentation of a surgical video to the standard of a practising urologist is feasible and practical for the RARP procedure. The assigning of temporal labels on a video labelling platform was also studied and proved feasible throughout the study period.

Two-directional two-dimensional fractional-order embedding canonical correlation analysis for multi-view dimensionality reduction and set-based video recognition

Set-based video recognition is an important application in practice, and many specialized approaches have been proposed. However, most of these methods either only use one kind of visual features for classification, or are sensitive to the noises and the number of training images when using several kinds of features, resulting in limited discrimination. To explore a possible solution to these issues, in this paper, a novel efficiently and effectively dimensionality reduction method, named two-directional two-dimensional fractional-order embedding canonical correlation analysis ((2D)2FECCA), is proposed. (2D)2FECCA borrows the idea of fractional-order embedding to correct the estimation of sample covariance matrices, which can significantly reduce the influence of noise disturbance and effectively utilizing the discriminative information from different view features. In addition, several set-based video recognition schemes are introduced to determine the labels of the test videos. Extensive experimental results on four familiar single image based databases and two video based benchmark databases demonstrate the effectiveness of the proposed method. These quantitative assessments reinforce the significance as well as the importance of embedding the proposed method in other intelligent systems application areas.

Ambiguousness-Aware State Evolution for Action Prediction

In this paper, we propose an ambiguousness-aware state evolution (AASE) method which represents the uncertainty of the input sequence and evolves the subsequent skeletons to generate a reasonable full-length sequence for action prediction. Unlike most existing methods that enforce partial sequences with the labels of full-length videos and ignore the semantic information of the subsequent action, we develop an evolution method by predicting the instructional actions and generating the reasonable candidate subsequent actions, so that the ambiguity of the full sequence’s label supervising for the partial actions can be effectively alleviated. Our method generates the rational subsequent actions under the instructional action class to complement the partially observed action sequence. We design two criteria for a rational generation: 1) the instruction of subsequent action keeps the semantic consistency with the observed sequence; 2) the generation sequence is satisfied with the distribution of the sequence of real data. Moreover, we design an uncertainty module to decide the instructional action class for the generation network. AASE predicts instructional actions with uncertainty learning and evolves different instructional actions by generating the subsequent skeletons, which find the most probable action to represent the partially observed action by learning the way of perceiving the tendency of the ongoing action. We conduct experiments on seven widely used action datasets: NTU-60, NTU-120, UCF101, UT-Interaction, BIT, PKU-MMD and HMDB51, and our experimental results clearly demonstrate that our method achieves very competitive performance with state-of-the-art.

Compressed video ensemble based pseudo-labeling for semi-supervised action recognition

Some recent studies have focused on deep learning based semi-supervised learning for action recognition. However, it is difficult to scale up their training because their input is RGB frames, the obtainment of which incurs computational and storage costs. In this paper, we propose a semi-supervised action recognition method that makes it easy to scale up the training by using features stored in compressed videos. Our method directly extracts multiple types of input features from compressed videos without any decoding and generates artificial labels of unlabeled videos through the ensembling of the predictions from these features. In addition to the standard supervised training on labeled videos, our models are trained to predict the artificial labels from strongly augmented features in unlabeled compressed videos. We show that our method is more efficient and achieves a better classification performance on some widely used datasets than conventional semi-supervised learning methods applying RGB frames.

RL-SSI Model: Adapting a Supervised Learning Approach to a Semi-Supervised Approach for Human Action Recognition

Generally, the action recognition task requires a vast amount of labeled data, which represents a time-consuming human annotation effort. To mitigate the dependency on labeled data, this study proposes Semi-Supervised and Iterative Reinforcement Learning (RL-SSI), which adapts a supervised approach that uses 100% labeled data to a semi-supervised and iterative approach using reinforcement learning for human action recognition in videos. The JIGSAWS and Breakfast datasets were used to evaluate the RL-SSI model, because they are commonly used in the action segmentation task. The same applies to the performance metrics used in this work-F-Score (F1) and Edit Score-which are commonly applied for such tasks. In JIGSAWS tests, we observed that the RL-SSI outperformed previously developed state-of-the-art techniques in all quantitative measures, while using only 65% of the labeled data. When analysing the Breakfast tests, we compared the effectiveness of RL-SSI with the results of the self-supervised technique called SSTDA. We have found that RL-SSI outperformed SSTDA with an accuracy of 66.44% versus 65.8%, but RL-SSI was surpassed by the F1@10 segmentation measure, which presented an accuracy of 67.33% versus 69.3% for SSTDA. Despite this, our experiment only used 55.8% of the labeled data, while SSTDA used 65%. We conclude that our approach outperformed equivalent supervised learning methods and is comparable to SSTDA, when evaluated on multiple datasets of human action recognition, proving to be an important innovative method to successfully building solutions to reduce the amount of fully labeled data, leveraging the work of human specialists in the task of data labeling of videos, and their respectives frames, for human action recognition, thus reducing the required resources to accomplish it.