Video Event Research Articles

Crowd Abnormal Event Detection Based on Crowd Density

Abstract This paper proposes a crowd abnormal event detection algorithm based on the change of crowd density. This method does not use the optical flow of the traditional model, thus the algorithm is fast. The algorithm first estimates the crowd density in the scene, then uses the change of crowd density as the feature representation of the crowd and constructs 3D feature blocks by adding time axis attributes. Finally, a single classifier is used to classify the 3D feature blocks for detecting crowd abnormal events in videos. Because the algorithm uses the change of crowd density in the scene as the feature representation of the crowd, the algorithm will not be disturbed by the motion of vehicles and other objects in the scene. The experimental results show that the proposed algorithm runs faster and has higher accuracy than the optical flow algorithm.

Examining the Reliability and Validity of Coding Perceived Force Severity and Bracing in the NHL Concussion Spotter Program.

Data obtained from the National Hockey League (NHL) have shown that a risk prediction model, including both visible signs and mechanisms of injury, improves the identification of possible concussion. However, only about half of concussions diagnosed by club medical staff in the NHL exhibit visible signs. At present, the NHL concussion spotter protocol does not include central league spotters' subjective judgments of the severity of forces associated with a direct hit to the head (perceived force severity [PFS]) or whether players brace before a hit (bracing). To examine the interrater reliability, preliminary validity, and association with concussion diagnosis of central league spotter determinations of PFS and bracing. Cross-sectional study. Video footage of 1071 events after a direct or indirect blow to the head were observed from the 2020-2021 and 2021-2022 NHL seasons. These events were classified into 4 groups: concussion with visible signs; concussion without visible signs; no concussion with visible signs; and no concussion without visible signs. A total of 50 events were randomly selected from the total events in each group. Then, 2 raters (NHL central league spotters) coded PFS for each of the 200 video events as low, medium, or high. Bracing was coded as no bracing, insufficient bracing, or full bracing. Interrater reliability was fair to moderate for the categorical and continuous ratings of both PFS (κ = 0.36 and 0.45, respectively) and bracing (κ = 0.40 and 0.49, respectively). There was no significant association between concussion diagnosis and either PFS (Z = 0.00, P = .99) or bracing (Z = 0.77, P = .44). Exploratory, post hoc analyses suggested a possible relationship between bracing and reduced concussion risk among a select subsample of events with no visible signs (r = -0.29, P < .01). The interrater reliability for PFS and for bracing was fair to moderate. Neither PFS nor bracing were significantly related to concussion diagnosis, but they were significantly associated with other visible signs and mechanisms of injury.

Clustering Aided Weakly Supervised Training to Detect Anomalous Events in Surveillance Videos.

Formulating learning systems for the detection of real-world anomalous events using only video-level labels is a challenging task mainly due to the presence of noisy labels as well as the rare occurrence of anomalous events in the training data. We propose a weakly supervised anomaly detection system that has multiple contributions including a random batch selection mechanism to reduce interbatch correlation and a normalcy suppression block (NSB) which learns to minimize anomaly scores over normal regions of a video by utilizing the overall information available in a training batch. In addition, a clustering loss block (CLB) is proposed to mitigate the label noise and to improve the representation learning for the anomalous and normal regions. This block encourages the backbone network to produce two distinct feature clusters representing normal and anomalous events. An extensive analysis of the proposed approach is provided using three popular anomaly detection datasets including UCF-Crime, ShanghaiTech, and UCSD Ped2. The experiments demonstrate the superior anomaly detection capability of our approach.

Automatic Identification of Game Stuttering via Gameplay Videos Analysis

Modern video games are extremely complex software systems and, as such, they might suffer from several types of post-release issues. A particularly insidious issue is constituted by drops in the frame rate ( i.e. , stuttering events), which might have a negative impact on the user experience. Stuttering events are frequently documented in the million of hours of gameplay videos shared by players on platforms such as Twitch or YouTube. From the developers’ perspective, these videos represent a free source of documented “testing activities”. However, especially for popular games, the quantity and length of these videos make impractical their manual inspection. We introduce HASTE, an approach for the automatic detection of stuttering events in gameplay videos that can be exploited to generate candidate bug reports. HASTE firstly splits a given video into visually coherent slices, with the goal of filtering-out those that not representing actual gameplay ( e.g ., navigating the game settings). Then, it identifies the subset of pixels in the video frames which actually show the game in action excluding additional elements on screen such as the logo of the YouTube channel, on-screen chats etc. In this way, HASTE can exploit state-of-the-art image similarity metrics to identify candidate stuttering events, namely subsequent frames being almost identical in the pixels depicting the game. We evaluate the different steps behind HASTE on a total of 105 videos showing that it can correctly extract video slices with a 76% precision, and can correctly identify the slices related to gameplay with a recall and precision higher than 77%. Overall, HASTE achieves 71% recall and 89% precision for the identification of stuttering events in gameplay videos.

Deep learning for surgical workflow analysis: a survey of progresses, limitations, and trends

Automatic surgical workflow analysis, which aims to recognize the ongoing surgical events in videos, is fundamental for developing context-aware computer-assisted systems. This paper reviews representative surgical workflow recognition algorithms based on deep learning, outlining their merits, limitations, and future research directions. The literature survey was performed on three large bibliographic databases, covering 67 lary sources, which were comparatively analyzed in terms of spatial feature modeling, spatio-temporal feature modeling, input pre-processing, regularization and post-processing algorithms, as well as learning strategies. Then, common public datasets and evaluation metrics for surgical workflow recognition are also described in detail. Finally, we discuss all literature from different perspectives, and point out the challenges, possible solutions and future trends. The need for more diverse and larger datasets, the potential of unsupervised and semi-supervised learning approaches, comprehensive and equitable metrics, establishing complete regulatory and data standards, and interoperability will be key challenges in translating models to clinical operating rooms. And we propose that surgical activity anticipation and employing large language model as training assistant are interesting research directions in surgical workflow analysis.

Video Anomaly Detection via self-supervised and spatio-temporal proxy tasks learning

Video Anomaly Detection (VAD) aims to identify events in videos that deviate from typical patterns. Given the scarcity of anomalous samples, previous research has primarily focused on learning regular patterns from datasets exclusively containing normal behaviors, and treating deviations from these patterns as anomalies. However, most of these methods are constrained by coarse-grained modeling approaches that renders them incapable of learning highly-discriminative features, which are necessary to effectively distinguish between the subtle differences between normal and abnormal behaviors. To better capture these features, we propose an innovative method. Initially, pseudo-anomalous samples for appearance and motion are generated through geometric transformations (2D rotations) and the scrambling of video sequences. Subsequently, a dual-branch network featuring spatio-temporal decoupling is proposed, in which the spatial and temporal branches each handle a specific proxy task. These tasks are designed to distinguish between normal and pseudo-anomalous samples, involving operations such as predicting patch-based 2D rotation angles and classifying video frame triplets as total-anomaly, left-anomaly, right-anomaly, and non-anomaly. Our approach employs an end-to-end training methodology, without relying on pre-trained models (except for the object detector). Evaluations on the UCSD Ped2, Avenue, and ShanghaiTech datasets show that our method achieved AUC scores of 99.1%, 91.9%, and 81.1%, respectively, demonstrating its effectiveness. The code is publicly accessible at the following link: https://spatio-temporal-tasks.

Safeguards-related event detection in surveillance video using semi-supervised learning approach

We develop a deep learning model employing a semi-supervised learning approach, which can detect automatically safeguards-related events in nuclear facility from surveillance video. Our model is designed after a comprehensive analysis of the trends in artificial intelligence-based models to identify abnormal events in video. Our model incorporates a reconstruction module and a prediction module independently. The reconstruction module is trained to generate video frames within a sliding window, while the prediction module is trained to predict future motion feature based on the motion features within the video frames in a sliding window. Each module utilizes an autoencoder with a memory module positioned between an encoder and an decoder of the autoencoder. We evaluate the model's performance using a benchmark dataset and a self-produced dataset obtained from facility related to pyroprocessing. Our model's performanace is comparable to or superior to that of the prevous models from the benchmark dataset analysis, and all the abnormal events can be detected without false positive error from the self-produced dataset analysis.

Distilling Privileged Knowledge for Anomalous Event Detection From Weakly Labeled Videos.

Weakly supervised video anomaly detection (WS-VAD) aims to identify the snippets involving anomalous events in long untrimmed videos, with solely text video-level binary labels. A typical paradigm among the existing text WS-VAD methods is to employ multiple modalities as inputs, e.g., RGB, optical flow, and audio, as they can provide sufficient discriminative clues that are robust to the diverse, complicated real-world scenes. However, such a pipeline has high reliance on the availability of multiple modalities and is computationally expensive and storage demanding in processing long sequences, which limits its use in some applications. To address this dilemma, we propose a privileged knowledge distillation (KD) framework dedicated to the WS-VAD task, which can maintain the benefits of exploiting additional modalities, while avoiding the need for using multimodal data in the inference phase. We argue that the performance of the privileged KD framework mainly depends on two factors: 1) the effectiveness of the multimodal teacher network and 2) the completeness of the useful information transfer. To obtain a reliable teacher network, we propose a text cross-modal interactive learning strategy and an anomaly normal discrimination loss, which target learning task-specific cross-modal features and encourage the separability of anomalous and normal representations, respectively. Furthermore, we design both representation- and text logits-level distillation loss functions, which force the unimodal student network to distill abundant privileged knowledge from the text well-trained multimodal teacher network, in a snippet-to-video fashion. Extensive experimental results on three public benchmarks demonstrate that the proposed privileged KD framework can train a lightweight yet effective detector, for localizing anomaly events under the supervision of video-level annotations.

Event-Oriented State Alignment Network for Weakly Supervised Temporal Language Grounding.

Weakly supervised temporal language grounding (TLG) aims to locate events in untrimmed videos based on natural language queries without temporal annotations, necessitating a deep understanding of semantic context across both video and text modalities. Existing methods often focus on simple correlations between query phrases and isolated video segments, neglecting the event-oriented semantic coherence and consistency required for accurate temporal grounding. This can lead to misleading results due to partial frame correlations. To address these limitations, we propose the Event-oriented State Alignment Network (ESAN), which constructs "start-event-end" semantic state sets for both textual and video data. ESAN employs relative entropy for cross-modal alignment through knowledge distillation from pre-trained large models, thereby enhancing semantic coherence within each modality and ensuring consistency across modalities. Our approach leverages vision-language models to extract static frame semantics and large language models to capture dynamic semantic changes, facilitating a more comprehensive understanding of events. Experiments conducted on two benchmark datasets demonstrate that ESAN significantly outperforms existing methods. By reducing false high correlations and improving the overall performance, our method effectively addresses the challenges posed by previous approaches. These advancements highlight the potential of ESAN to improve the precision and reliability of temporal language grounding tasks.

Transformer and Adaptive Threshold Sliding Window for Improving Violence Detection in Videos.

This paper presents a comprehensive approach to detect violent events in videos by combining CrimeNet, a Vision Transformer (ViT) model with structured neural learning and adversarial regularization, with an adaptive threshold sliding window model based on the Transformer architecture. CrimeNet demonstrates exceptional performance on all datasets (XD-Violence, UCF-Crime, NTU-CCTV Fights, UBI-Fights, Real Life Violence Situations, MediEval, RWF-2000, Hockey Fights, Violent Flows, Surveillance Camera Fights, and Movies Fight), achieving high AUC ROC and AUC PR values (up to 99% and 100%, respectively). However, the generalization of CrimeNet to cross-dataset experiments posed some problems, resulting in a 20-30% decrease in performance, for instance, training in UCF-Crime and testing in XD-Violence resulted in 70.20% in AUC ROC. The sliding window model with adaptive thresholding effectively solves these problems by automatically adjusting the violence detection threshold, resulting in a substantial improvement in detection accuracy. By applying the sliding window model as post-processing to CrimeNet results, we were able to improve detection accuracy by 10% to 15% in cross-dataset experiments. Future lines of research include improving generalization, addressing data imbalance, exploring multimodal representations, testing in real-world applications, and extending the approach to complex human interactions.

Feature fusion-based video summarization using SegNetSN

Feature fusion-based video summarization using SegNetSN

Identifying intraoperative events in a simulated laparotomy video: a multinational study of inattentional blindness among anesthesiologists.

Medical errors may be occasionally explained by inattentional blindness (IB), i.e., failing to notice an event/object that is in plain sight. We aimed to determine whether age/experience, restfulness/fatigue, and previous exposure to simulation education may affect IB in the anesthetic/surgical setting. In this multicentre/multinational study, a convenience sample of 280 anesthesiologists watched an attention-demanding video of a simulated trauma patient undergoing laparotomy and (independently/anonymously) recorded the abnormalities they noticed. The video contained four expected/common abnormalities (hypotension, tachycardia, hypoxia, hypothermia) and two prominently displayed unexpected/rare events (patient's head movement, leaky central venous line). We analyzed the participants' ability to notice the expected/unexpected events (primary outcome) and the proportion of expected/unexpected events according to age group and prior exposure to simulation education (secondary outcomes). Anesthesiologists across all ages noticed fewer unexpected/rare events than expected/common ones. Overall, younger anesthesiologists missed fewer common events than older participants did (P = 0.02). There was no consistent association between age and perception of unexpected/rare events (P = 0.28), although the youngest cohort (< 30yr) outperformed the other age groups. Prior simulation education did not affect the proportion of misses for the unexpected/rare events but was associated with fewer misses for the expected/common events. Self-perceived restfulness did not impact perception of events. Anesthesiologists noticed fewer unexpected/rare clinical events than expected/common ones in an attention-demanding video of a simulated trauma patient, in keeping with IB. Prior simulation training was associated with an improved ability to notice anticipated/expected events, but did not reduce IB. Our findings may have implications for understanding medical mishaps, and efforts to improve situational awareness, especially in acute perioperative and critical care settings.

Optimal object stitching and video condensation-aided improved YOLOv5-oriented anomaly object detection framework

Recent advances in intelligent surveillance to boost public security have increased interest in anomaly location and identification among computer vision researchers in congested situations. Unrestricted video monitoring is required due to its extensive use in private and public spheres. Particularly, research on finding anomalies in surveillance videos has drawn much interest. Video condensation is an efficient method for quickly scanning and retrieving lengthy surveillance films. Reordering the video events in the spatial or temporal domain seeks to condense large video sequences into their equivalent compact video representation. Additionally, if a delay occurs in anomaly detection, it leads to time consumption for rectifying it. Therefore, the core aim of this task is to reduce the time consumption of detecting the instance of video anomaly detection faster by introducing the optimal object stitching through the deep learning-based video condensation concept. The steps include video condensation and deep learning-based anomaly detection. Initially, the necessary video for anomaly detection is gathered from the datasets. The gathered video is converted into frames and is used for object detection in each frame using Improved YOLOv5 (I-YOLOv5). Then, the optimal frame selection is performed using the hybrid Fitness of Wild Geese and Path Finder (FWGPF) for stitching the detected objects into the selected optimal frames, thus compressing the relevant frames. In the second phase, the condensed (compressed) frames are utilized for determining the anomaly frames with the support of Parameter Tuning in Dilated MobileNet (PTDM), and further, the anomaly objects are detected on the classified condensed frames based on the I-YOLOv5. Here, the tuning of constraint is done in the anomaly detection phase by the same hybrid FWGPF to reduce the time consumption. The experimental analysis of the developed model is performed with baseline algorithms and detection techniques to confirm the developed model’s efficiency.

A Deep Learning Framework for Monitoring Audience Engagement in Online Video Events

This paper introduces a deep learning methodology for analyzing audience engagement in online video events. The proposed deep learning framework consists of six layers and starts with keyframe extraction from the video stream and the participants’ face detection. Subsequently, the head pose and emotion per participant are estimated using the HopeNet and JAA-Net deep architectures. Complementary to video analysis, the audio signal is also processed using a neural network that follows the DenseNet-121 architecture. Its purpose is to detect events related to audience engagement, including speech, pauses, and applause. With the combined analysis of video and audio streams, the interest and attention of each participant are inferred more accurately. An experimental evaluation is performed on a newly generated dataset consisting of recordings from online video events, where the proposed framework achieves promising results. Concretely, the F1 scores were 79.21% for interest estimation according to pose, 65.38% for emotion estimation, and 80% for sound event detection. The proposed framework has applications in online educational events, where it can help tutors assess audience engagement and comprehension while hinting at points in their lectures that may require further clarification. It is effective for video streaming platforms that want to provide video recommendations to online users according to audience engagement.

Deep learning based anomaly detection in real-time video

AbstractMany security cameras have been put up in places like airports, roads, and banks for the safety of these public places. These cameras make a lot of video data, and most security camera recordings are only ever seen when something strange happens. This means that monitoring has to be done by people, which is time-consuming and often wrong, so automatic ways of monitoring have to be used. In this paper, we propose a system that automatically detects irregular events in videos based on the integration of Inflated 3D Convolution Network (I3D-ResNet50) and deep Multiple Instance Learning (MIL). This system considers both regular and unusual videos as negative and positive packets, respectively. Each video snippet is a case of that packet. An anomaly score is generated for each video snippet using a fully connected Neural Network (NN). After processing videos, we used an I3D-ResNet50 to extract features after applying 10-crop augmentations to the UCF-101 dataset that contains 130 GB of videos with 13 abnormal events such as fighting, stealing, abuse, etc., as well as normal events. Our experimental results show that the AUC is 82.85% with only 10,000 iterations compared with other approaches. This means that our model is better at spotting anomalies in real-time videos.

Modeling and Performance Analysis of a Notification-Based Method for Processing Video Queries on the Fly

With the rapid growth of video data, the search for content and events in videos is becoming increasingly relevant, and many challenges arise. Various approaches have been proposed to deal with many issues. However, many open questions are still related to computational cost and latency, especially for real-time applications. Considering the need for new and efficient solutions, the so-called NOP (Notification Oriented Paradigm) could be a suitable alternative. NOP introduced a new way of thinking and developing software in which small collaborative entities perform fact execution and logical decision processing based on precise notifications. Following these concepts and practical tools, this paper proposes a new querying processing method based on NOP, focusing on search and matching in a continuous flow context. Experiments on a labeled dataset demonstrated the suitability of the proposed method for low-latency processing with polynomial complexity. The results are better than the state of the art, which works at exponential cost.

Local feature‐based video captioning with multiple classifier and CARU‐attention

AbstractVideo captioning aims to identify multiple objects and their behaviours in a video event and generate captions for the current scene. This task aims to generate a detailed description of the current video in real‐time using natural language, which requires deep learning to analyze and determine the relationships between interesting objects in the frame sequence. In practice, existing methods typically involve detecting objects in the frame sequence and then generating captions based on features extracted through object coverage locations. Therefore, the results of caption generation are highly dependent on the performance of object detection and identification. This work proposes an advanced video captioning approach that works in adaptively and effectively addresses the interdependence between event proposals and captions. Additionally, an attention‐based multimodel framework is introduced to capture the main context from the frame and sound in the video scene. Also, an intermediate model is presented to collect the hidden states captured from the input sequence, which performs to extract the main features and implicitly produce multiple event proposals. For caption prediction, the proposed method employs the CARU layer with attention consideration as the primary RNN layer for decoding. Experimental results showed that the proposed work achieves improvements compared to the baseline method and also better performance compared to other state‐of‐the‐art models on the ActivityNet dataset, presenting competitive results in the tasks of video captioning.

Basketball Tactics Analysis Based on Improved Openpose Algorithm and Its Application

The use of information technology has currently resulted in a change in people's production and lifestyles as well as a catalyst for the sports industry's evolution. Currently, there is a growing use of digitization in the sports industry's basketball realm. Using 2D posture estimation, a novel lightweight deep learning architecture (LDLA) is built to accomplish the automatic analysis of basketball game footage. We offer a real-time method to identify numerous people's 2D poses in a video. Additionally, it examines how 2D pose estimation might be used to analyse basketball shooting videos. First, group and global motion features are extracted to represent semantic events. A full basketball game video is broken down into three stages: clip-based segmentation, classification of semantics occurrences utilizing audio and video, and temporal sequence characteristics employing GRU CNN. Video is first processed to create some basic video categorization and segmentation using the visual, movement, and auditory data. Further use of domain knowledge is made to find important events in the basketball video. Shot and image threshold recognition algorithms for videos have used both optical and kinematic prediction information; scene classification comes next. The placements of probable semantic events, such as "fouling" and "shooting at the basket," are then discovered by comparing the multidimensional data with supplementary domain expertise. Experimental results demonstrate the proposed LDLA method achieves 99.6% of accuracy, 93.2% of precision, 93.5% of recall and 89.5% of F1-score.

Video Event Extraction with Multi-View Interaction Knowledge Distillation

Video event extraction (VEE) aims to extract key events and generate the event arguments for their semantic roles from the video. Despite promising results have been achieved by existing methods, they still lack an elaborate learning strategy to adequately consider: (1) inter-object interaction, which reflects the relation between objects; (2) inter-modality interaction, which aligns the features from text and video modality. In this paper, we propose a Multi-view Interaction with knowledge Distillation (MID) framework to solve the above problems with the Knowledge Distillation (KD) mechanism. Specifically, we propose the self-Relational KD (self-RKD) to enhance the inter-object interaction, where the relation between objects is measured by distance metric, and the high-level relational knowledge from the deeper layer is taken as the guidance for boosting the shallow layer in the video encoder. Meanwhile, to improve the inter-modality interaction, the Layer-to-layer KD (LKD) is proposed, which integrates additional cross-modal supervisions (i.e., the results of cross-attention) with the textual supervising signal for training each transformer decoder layer. Extensive experiments show that without any additional parameters, MID achieves the state-of-the-art performance compared to other strong methods in VEE.

HiFormer: Hierarchical transformer for grounded situation recognition

The prevalence of monitoring video is critical to public safety, but existing Object Detection and Action Recognition models are overwhelmed by camera operators, unable to identify relevant events. In light of this, Grounding Situation Recognition (GSR) provides a practical solution to recognize the events in a surveillance video. That is, GSR can identify the noun entities (e.g., humans) and their actions (e.g., driving), and provide grounding frames for involved entities. Compared with Action Recognition and Object Detection, GSR is more in line with human cognitive habits, better allowing law enforcement agencies to understand the predictions. However, the crucial issue with most existing frameworks is the neglect of verb ambiguity, that is, superficially similar verbs but have distinct meanings (e.g. buying v.s. giving). Many existing works propose a two-stage model, which first blindly predicts the verb, and then uses this verb information to predict semantic roles. These frameworks ignore the importance of noun information during verb prediction, making them susceptible to misidentifications. To address this problem and better discern between ambiguous verbs, we propose HiFormer, a novel hierarchical transformer framework with direct and comprehensive consideration of similar verbs for each image, to more accurately identify the salient verb, semantic roles, and the grounding frames. Compared with the state-of-the-art models in Grounded Situation Recognition (SituFormer and CoFormer), HiFormer shows an advantage of over 35% and 20% on the Top-1 and Top-5 verb accuracy respectively, as well as 13% on the Top-1 Noun accuracy.