Video Snippets Research Articles

Leveraging Multi-Modality and Enhanced Temporal Networks for Robust Violence Detection

In this paper, we present a novel model that enhances performance by extending the dual-modality TEVAD model—originally leveraging visual and textual information—into a multi-modal framework that integrates visual, audio, and textual data. Additionally, we refine the multi-scale temporal network (MTN) to improve feature extraction across multiple temporal scales between video snippets. Using the XD-Violence dataset, which includes audio data for violence detection, we conduct experiments to evaluate various feature fusion methods. The proposed model achieves an average precision (AP) of 83.9%, surpassing the performance of single-modality approaches (visual: 73.9%, audio: 67.1%, textual: 29.9%) and dual-modality approaches (visual + audio: 78.8%, visual + textual: 78.5%). These findings demonstrate that the proposed model outperforms models based on the original MTN and reaffirm the efficacy of multi-modal approaches in enhancing violence detection compared to single- or dual-modality methods.

Multimodal and multiscale feature fusion for weakly supervised video anomaly detection

Weakly supervised video anomaly detection aims to detect anomalous events with only video-level labels. In the absence of boundary information for anomaly segments, most existing methods rely on multiple instance learning. In these approaches, the predictions for unlabeled video snippets are guided by the classification of labeled untrimmed videos. However, these methods do not account for issues such as video blur and visual occlusion, which can hinder accurate anomaly detection. To address these issues, we propose a novel weakly supervised video anomaly detection method that fuses multimodal and multiscale features. Firstly, RGB and optical flow snippets are input into pre-trained I3D to extract appearance and motion features. Then, we introduce an Attention De-redundancy (AD) module, which employs an attention mechanism to filter out task-irrelevant redundancy in these appearance and motion features. Next, to mitigate the effects of video blurring and visual occlusion, we propose a Multi-scale Feature Learning module. This module captures long-term and short-term temporal dependencies among video snippets to provide global and local guidance for blurred or occluded video snippets. Finally, to effectively utilize the discriminative features of different modalities, we propose an Adaptive Feature Fusion module. This module adaptively fuses appearance and motion features based on their respective feature weights. Extensive experimental results demonstrate that our proposed method outperforms mainstream unsupervised and weakly supervised methods in terms of AUC. Specifically, our proposed method achieves 97.00% AUC and 85.31% AUC on two benchmark datasets, i.e., ShanghaiTech and UCF-Crime, respectively.

A new approach to video summary generation

This paper describes a video summary approach for an actual video file that provides a rapid way to assess its information. This is a difficult task to identify relevant or informative sections. When the original video demands knowledge of its content, we build a video summary via extracting in-depth characteristics of each original video. Video summarization deals with the generation of a condensed version of the original video by including meaningful frames or segments while eliminating redundant information provides a concise and accurate representation of the original video clips by providing the most representative summary with increased interest through the application of a matching function between the current and prior frames. The primary goal of this work is to give a strategy for recording video clips using security cameras and converting them into a short video to provide a clear analysis of the video by reducing excess and extracting the key video contents into the real video. In a lengthy video clip, viewers may not have enough time to see the full thing. The viewer may be interested in seeing a specific topic under the label "Important". It may be composed of pieces that may be shared on social media and communicate the semantics of long videos, particularly video snippets that must be submitted to Internet. To browse, will be needed a network with adequate bandwidth. That is, non-affiliated video streaming, indexing, and retrieval of video content makes it valuable to viewers.

Learning a multi-cluster memory prototype for unsupervised video anomaly detection

In recent years, there has been rapid development in video anomaly detection (VAD). The previous methods ignored the differences between normal videos and only emphasized learning the commonalities of normal videos. In order to improve the performance of anomaly detection, we delve into the spatial distribution of normal video features and utilize their differences for clustering, leading to more minor reconstruction errors for normal videos and more significant reconstruction errors for abnormal videos. To achieve this goal, we introduce a Multi-Cluster Memory Prototype framework (MCMP) for VAD, which explores the coarse-grained and fine-grained information from video snippet features simultaneously to learn a memory prototype, thereby significantly improving the ability to discriminate abnormal events in complex scenes. First, a video feature clustering method that employs contrastive learning is introduced to group samples sharing similar fine-grained features. Second, the memory mechanism is used to capture the feature distribution of normal samples. Lastly, the Gaussian filter feature transformation method is introduced to make normal and abnormal features more distinguishable. The frame level AUC of MCMP on ShanghaiTech and UCF-Crime benchmark datasets has increased by 1.26% and 0.45% compared to state-of-the-art methods. Our code is publicly available athttps://github.com/WuIkun5658/MCMP.

Cross-view motion consistent self-supervised video inter-intra contrastive for action representation understanding

Self-supervised contrastive learning draws on power representational models to acquire generic semantic features from unlabeled data, and the key to training such models lies in how accurately to track motion features. Previous video contrastive learning methods have extensively used spatially or temporally augmentation as similar instances, resulting in models that are more likely to learn static backgrounds than motion features. To alleviate the background shortcuts, in this paper, we propose a cross-view motion consistent (CVMC) self-supervised video inter-intra contrastive model to focus on the learning of local details and long-term temporal relationships. Specifically, we first extract the dynamic features of consecutive video snippets and then align these features based on multi-view motion consistency. Meanwhile, we compare the optimized dynamic features for instance comparison of different videos and local spatial fine-grained with temporal order in the same video, respectively. Ultimately, the joint optimization of spatio-temporal alignment and motion discrimination effectively fills the challenges of the missing components of instance recognition, spatial compactness, and temporal perception in self-supervised learning. Experimental results show that our proposed self-supervised model can effectively learn visual representation information and achieve highly competitive performance compared to other state-of-the-art methods in both action recognition and video retrieval tasks.

Fusing crops representation into snippet via mutual learning for weakly supervised surveillance anomaly detection

AbstractIn recent years, the challenge of detecting anomalies in real‐world surveillance videos using weakly supervised data has emerged. Traditional methods, utilising multi‐instance learning (MIL) with video snippets, struggle with background noise and tend to overlook subtle anomalies. To tackle this, the authors propose a novel approach that crops snippets to create multiple instances with less noise, separately evaluates them and then fuses these evaluations for more precise anomaly detection. This method, however, leads to higher computational demands, especially during inference. Addressing this, our solution employs mutual learning to guide snippet feature training using these low‐noise crops. The authors integrate multiple instance learning (MIL) for the primary task with snippets as inputs and multiple‐multiple instance learning (MMIL) for an auxiliary task with crops during training. The authors’ approach ensures consistent multi‐instance results in both tasks and incorporates a temporal activation mutual learning module (TAML) for aligning temporal anomaly activations between snippets and crops, improving the overall quality of snippet representations. Additionally, a snippet feature discrimination enhancement module (SFDE) refines the snippet features further. Tested across various datasets, the authors’ method shows remarkable performance, notably achieving a frame‐level AUC of 85.78% on the UCF‐Crime dataset, while reducing computational costs.

Using Authentic Video Snippets for Developing Non-Major Students' EFL Listening Skills and Growth Language Mindset

Using Authentic Video Snippets for Developing Non-Major Students' EFL Listening Skills and Growth Language Mindset

POTLoc: Pseudo-label Oriented Transformer for point-supervised temporal Action Localization

This paper tackles the challenge of point-supervised temporal action detection, wherein only a single frame is annotated for each action instance in the training set. Most of the current methods, hindered by the sparse nature of annotated points, struggle to effectively represent the continuous structure of actions or the inherent temporal and semantic dependencies within action instances. Consequently, these methods frequently learn merely the most distinctive segments of actions, leading to the creation of incomplete action proposals. This paper proposes POTLoc, a Pseudo-label Oriented Transformer for weakly-supervised Action Localization utilizing only point-level annotation. POTLoc is designed to identify and track continuous action structures via a self-training strategy. The base model begins by generating action proposals solely with point-level supervision. These proposals undergo refinement and regression to enhance the precision of the estimated action boundaries, which subsequently results in the production of ‘pseudo-labels’ to serve as supplementary supervisory signals. The architecture of the model integrates a transformer with a temporal feature pyramid to capture video snippet dependencies and model actions of varying duration. The pseudo-labels, providing information about the coarse locations and boundaries of actions, assist in guiding the transformer for enhanced learning of action dynamics. POTLoc outperforms the state-of-the-art point-supervised methods on THUMOS’14 and ActivityNet-v1.2 datasets.

Deep learning based anomaly detection in real-time video

AbstractMany security cameras have been put up in places like airports, roads, and banks for the safety of these public places. These cameras make a lot of video data, and most security camera recordings are only ever seen when something strange happens. This means that monitoring has to be done by people, which is time-consuming and often wrong, so automatic ways of monitoring have to be used. In this paper, we propose a system that automatically detects irregular events in videos based on the integration of Inflated 3D Convolution Network (I3D-ResNet50) and deep Multiple Instance Learning (MIL). This system considers both regular and unusual videos as negative and positive packets, respectively. Each video snippet is a case of that packet. An anomaly score is generated for each video snippet using a fully connected Neural Network (NN). After processing videos, we used an I3D-ResNet50 to extract features after applying 10-crop augmentations to the UCF-101 dataset that contains 130 GB of videos with 13 abnormal events such as fighting, stealing, abuse, etc., as well as normal events. Our experimental results show that the AUC is 82.85% with only 10,000 iterations compared with other approaches. This means that our model is better at spotting anomalies in real-time videos.

Weakly Supervised Video Anomaly Detection via Self-Guided Temporal Discriminative Transformer.

Weakly supervised video anomaly detection is generally formulated as a multiple instance learning (MIL) problem, where an anomaly detector learns to generate frame-level anomaly scores under the supervision of MIL-based video-level classification. However, most previous works suffer from two drawbacks: 1) they lack ability to model temporal relationships between video segments and 2) they cannot extract sufficient discriminative features to separate normal and anomalous snippets. In this article, we develop a weakly supervised temporal discriminative (WSTD) paradigm, that aims to leverage both temporal relation and feature discrimination to mitigate the above drawbacks. To this end, we propose a transformer-styled temporal feature aggregator (TTFA) and a self-guided discriminative feature encoder (SDFE). Specifically, TTFA captures multiple types of temporal relationships between video snippets from different feature subspaces, while SDFE enhances the discriminative powers of features by clustering normal snippets and maximizing the separability between anomalous snippets and normal centers in embedding space. Experimental results on three public benchmarks indicate that WSTD outperforms state-of-the-art unsupervised and weakly supervised methods, which verifies the superiority of the proposed method.

TransGANomaly: Transformer based Generative Adversarial Network for Video Anomaly Detection

Video anomaly detection aims to identify a set of abnormal events in videos. Deep reconstruction and prediction-based models have been employed to detect anomalies. Deep reconstruction models sometimes recreate the abnormal events along with the normal ones. However, the prediction-based approaches have demonstrated encouraging results. This paper presents a video vision transformer (ViViT) based generative adversarial network (GAN), TransGANomaly, a novel approach for detecting anomalies. The proposed framework is a video frame predictor and trained only on normal video data adversarially. The generator of the GAN is a ViViT network that receives 3D input tokens from the video snippets. The generator aims to predict the future frame based on past sequences. After that, the predicted and original frames are given to the model’s discriminator for binary classification. Extensive experiments have been performed on UCSD Pedestrian, CUHK Avenue, and ShanghiaTech datasets to validate the efficacy of the proposed method.

Action Segmentation Through Self-Supervised Video Features and Positional-Encoded Embeddings

Action segmentation consists of temporally segmenting a video and labeling each segmented interval with a specific action label. In this work, we propose a novel action segmentation method that requires no initial video analysis and no annotated data. Our proposal involves extracting features from videos using several pre-trained deep-learning models, including spatiotemporal and self-supervised methods. Data is then transformed using a positional encoder, and finally, a clustering algorithm is applied, where each produced cluster presumably corresponds to a different single and distinguishable action. For self-supervised features, we explored DINO, and for spatiotemporal features, we investigated I3D and SlowFast methods. Moreover, two different clustering algorithms (FINCH and KMeans) were investigated, and we also explored how varying the length of video snippets that generate the feature vectors affected the quality of the segmentation. Experiments show that our method produces competitive results on the Breakfast and INRIA Instructional Videos dataset benchmarks. Our best result was produced using a composition of self-supervised features generated by DINO, FINCH clustering, and positional encoding.

Branding on Social Media Platforms As a Solution For Mixed Procession Music Groups In The City Of Padang, West Sumatra, Indonesia

Pancaragam, the only active music group in West Sumatra Province, Indonesia, holds significant potential but has yet to gain widespread recognition. Similar to Tanjidor groups in Betawi Kalimantan, such as Pontianak and Pemangkat, Pancaragam faces challenges in expanding its reach and economic impact on its musicians. To address this, we offer strategic solutions focused on enhancing visibility through social media platforms. This proposal includes the implementation of management patterns that align with the characteristics of each social media platform. Firstly, a profound understanding is needed on how to manage and leverage the specific features of platforms such as Instagram, Facebook, and YouTube. This involves the use of compelling visual content, such as video snippets from their performances, as well as active engagement with fans through comments and direct messaging. Furthermore, the implementation of creative digital campaign strategies can elevate Pancaragam's online presence. Contests or music challenges on these platforms can encourage community participation, while special offers or discounts for live performances can attract a broader audience. Collaborating with local influencers or social media users with an interest in traditional arts and culture can also be an effective way to expand networks and reach new audiences. With a combination of focused management and digital marketing strategies, Pancaragam can gain broader recognition, enhance their online presence, and ultimately provide a positive economic impact for its musicians.

Instrument-Tissue Interaction Detection Framework for Surgical Video Understanding.

Instrument-tissue interaction detection task, which helps understand surgical activities, is vital for constructing computer-assisted surgery systems but with many challenges. Firstly, most models represent instrument-tissue interaction in a coarse-grained way which only focuses on classification and lacks the ability to automatically detect instruments and tissues. Secondly, existing works do not fully consider relations between intra- and inter-frame of instruments and tissues. In the paper, we propose to represent instrument-tissue interaction as 〈 instrument class, instrument bounding box, tissue class, tissue bounding box, action class 〉 quintuple and present an Instrument-Tissue Interaction Detection Network (ITIDNet) to detect the quintuple for surgery videos understanding. Specifically, we propose a Snippet Consecutive Feature (SCF) Layer to enhance features by modeling relationships of proposals in the current frame using global context information in the video snippet. We also propose a Spatial Corresponding Attention (SCA) Layer to incorporate features of proposals between adjacent frames through spatial encoding. To reason relationships between instruments and tissues, a Temporal Graph (TG) Layer is proposed with intra-frame connections to exploit relationships between instruments and tissues in the same frame and inter-frame connections to model the temporal information for the same instance. For evaluation, we build a cataract surgery video (PhacoQ) dataset and a cholecystectomy surgery video (CholecQ) dataset. Experimental results demonstrate the promising performance of our model, which outperforms other state-of-the-art models on both datasets.

How Clinicians Decide? Exploring Complexity of Antibiotic Prescribing in Emergency Departments Using Video-Reflexive Ethnography.

Antibiotic overprescribing is a global issue that significantly contributes to increased antimicrobial resistance. Strengthening antimicrobial prescribing practices should be considered a priority. The emergency department (ED) represents a setting where antibiotics are frequently prescribed, but the determinants that influence prescribing choices are complex and multifaceted. We conducted an exploratory qualitative study to investigate the contextual factors that influence antibiotic prescribing choices among clinicians in the ED. The study employed video-reflexive ethnography (VRE) to capture prospective clinical decision-making in situated practice. Data collection involved fieldwork observations, video observations, and delivery of facilitated group reflexive sessions, where clinicians viewed a selection of recorded video snippets relating to antibiotic prescribing. Study was conducted across two EDs within the same health service in Australia. A total of 29 clinical conversations focusing on antibiotic prescribing were recorded. Additionally, 34 clinicians participated in group reflexive sessions. Thematic analysis from the transcribed data yielded four themes: 'importance of clinical judgment', 'usability of prescribing guidelines', 'managing patient expectations', and 'context-dependent disruptions'. Our findings provide insights into the challenges faced by clinicians in navigating complex ED environment, utilising electronic decision-support tools and engaging in discussions about patient treatments with senior clinicians. The findings also indicate that VRE is useful in visualising full complexity of the ED setting, and in initiating meaningful discussions among clinical teams. Integrating the use of VRE in everyday clinical settings can potentially facilitate the implementation of pragmatic solutions for delivering effective antibiotic stewardship practices.

CNN-ViT Supported Weakly-Supervised Video Segment Level Anomaly Detection.

Video anomaly event detection (VAED) is one of the key technologies in computer vision for smart surveillance systems. With the advent of deep learning, contemporary advances in VAED have achieved substantial success. Recently, weakly supervised VAED (WVAED) has become a popular VAED technical route of research. WVAED methods do not depend on a supplementary self-supervised substitute task, yet they can assess anomaly scores straightway. However, the performance of WVAED methods depends on pretrained feature extractors. In this paper, we first address taking advantage of two pretrained feature extractors for CNN (e.g., C3D and I3D) and ViT (e.g., CLIP), for effectively extracting discerning representations. We then consider long-range and short-range temporal dependencies and put forward video snippets of interest by leveraging our proposed temporal self-attention network (TSAN). We design a multiple instance learning (MIL)-based generalized architecture named CNN-ViT-TSAN, by using CNN- and/or ViT-extracted features and TSAN to specify a series of models for the WVAED problem. Experimental results on publicly available popular crowd datasets demonstrated the effectiveness of our CNN-ViT-TSAN.

Snipper: A Spatiotemporal Transformer for Simultaneous Multi-Person 3D Pose Estimation Tracking and Forecasting on a Video Snippet

Multi-person pose understanding from RGB videos involves three complex tasks: pose estimation, tracking and motion forecasting. Intuitively, accurate multi-person pose estimation facilitates robust tracking, and robust tracking builds crucial history for correct motion forecasting. Most existing works either focus on a single task or employ multi-stage approaches to solving multiple tasks separately, which tends to make sub-optimal decision at each stage and also fail to exploit correlations among the three tasks. In this paper, we propose Snipper, a unified framework to perform multi-person 3D pose estimation, tracking, and motion forecasting simultaneously in a single stage. We propose an efficient yet powerful deformable attention mechanism to aggregate spatiotemporal information from the video snippet. Building upon this deformable attention, a video transformer is learned to encode the spatiotemporal features from the multi-frame snippet and to decode informative pose features for multi-person pose queries. Finally, these pose queries are regressed to predict multi-person pose trajectories and future motions in a single shot. In the experiments, we show the effectiveness of Snipper on three challenging public datasets where our generic model rivals specialized state-of-art baselines for pose estimation, tracking, and forecasting. Code is available at https://github.com/JimmyZou/Snipper.

Egocentric Early Action Prediction via Multimodal Transformer-Based Dual Action Prediction

Egocentric early action prediction, which aims to recognize the on-going action in the video captured in the first-person view as early as possible before the action is fully executed, is a new yet challenging task due to the limited partial video input. Pioneer studies focused on solving this task with LSTMs as the backbone and simply compiling the observed video segment and unobserved video segment into a single vector, which hence suffer from two key limitations: lack the non-sequential relation modeling with the video snippet sequence and the correlation modeling between the observed and unobserved video segment. To address these two limitations, in this paper, we propose a novel multimodal TransfoRmer-based duAl aCtion prEdiction (mTRACE) model for the task of egocentric early action prediction, which consists of two key modules: the early (observed) segment action prediction module and the future (unobserved) segment action prediction module. Both modules take Transformer encoders as the backbone for encoding all the potential relations among the input video snippets, and involve several single-modal and multi-modal classifiers for comprehensive supervision. Different from previous work, each of the two modules outputs two multi-modal feature vectors: one for encoding the current input video segment, and the other one for predicting the missing video segment. For optimization, we design a two-stage training scheme, including the mutual enhancement stage and end-to-end aggregation stage. The former stage alternatively optimizes the two action prediction modules, where the correlation between the observed and unobserved video segment is modeled with a consistency regularizer, while the latter seamlessly aggregates the two modules to fully utilize the capacity of the two modules. Extensive experiments have demonstrated the superiority of our proposed model. We have released the codes and the corresponding parameters to benefit other researchers <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .

ContextLoc++: A Unified Context Model for Temporal Action Localization.

Effectively tackling the problem of temporal action localization (TAL) necessitates a visual representation that jointly pursues two confounding goals, i.e., fine-grained discrimination for temporal localization and sufficient visual invariance for action classification. We address this challenge by enriching the local, global and multi-scale contexts in the popular two-stage temporal localization framework. Our proposed model, dubbed ContextLoc++, can be divided into three sub-networks: L-Net, G-Net, and M-Net. L-Net enriches the local context via fine-grained modeling of snippet-level features, which is formulated as a query-and-retrieval process. Furthermore, the spatial and temporal snippet-level features, functioning as keys and values, are fused by temporal gating. G-Net enriches the global context via higher-level modeling of the video-level representation. In addition, we introduce a novel context adaptation module to adapt the global context to different proposals. M-Net further fuses the local and global contexts with multi-scale proposal features. Specially, proposal-level features from multi-scale video snippets can focus on different action characteristics. Short-term snippets with fewer frames pay attention to the action details while long-term snippets with more frames focus on the action variations. Experiments on the THUMOS14 and ActivityNet v1.3 datasets validate the efficacy of our method against existing state-of-the-art TAL algorithms.

Automated Motor Tic Detection: A Machine Learning Approach.

Video-based tic detection and scoring is useful to independently and objectively assess tic frequency and severity in patients with Tourette syndrome. In trained raters, interrater reliability is good. However, video ratings are time-consuming and cumbersome, particularly in large-scale studies. Therefore, we developed two machine learning (ML) algorithms for automatic tic detection. The aim of this study was to evaluate the performances of state-of-the-art ML approaches for automatic video-based tic detection in patients with Tourette syndrome. We used 64 videos of n = 35 patients with Tourette syndrome. The data of six subjects (15 videos with ratings) were used as a validation set for hyperparameter optimization. For the binary classification task to distinguish between tic and no-tic segments, we established two different supervised learning approaches. First, we manually extracted features based on landmarks, which served as input for a Random Forest classifier (Random Forest). Second, a fully automated deep learning approach was used, where regions of interest in video snippets were input to a convolutional neural network (deep neural network). Tic detection F1 scores (and accuracy) were 82.0% (88.4%) in the Random Forest and 79.5% (88.5%) in the deep neural network approach. ML algorithms for automatic tic detection based on video recordings are feasible and reliable and could thus become a valuable assessment tool, for example, for objective tic measurements in clinical trials. ML algorithms might also be useful for the differential diagnosis of tics. © 2023 The Authors. Movement Disorders published by Wiley Periodicals LLC on behalf of International Parkinson and Movement Disorder Society.