Frame Level Research Articles

Pose-oriented scene-adaptive matching for abnormal event detection

For intelligent surveillance systems, abnormal event detection automatically analyses surveillance video sequences and detects abnormal objects or unusual human actions at the frame level. Due to the lack of labelled data, most approaches are semi-supervised based on reconstruction or prediction methods. However, these methods may not generalize well to unseen scene contexts. To address this issue, we present a novel self and mutual scene-adaptive matching method for abnormal event detection. In the framework, we propose synergistic pose estimation and object detection, which effectively integrates human pose and object detection information to improve pose estimation accuracy. Then, the poses are resized to reduce the spatial distance between the source and target domains. The improved pose sequences are further fed into a spatio-temporal graph convolutional network to extract the geometric features. Finally, the features are embedded in a clustering layer to classify action types and compute normality scores. The training data is taken from the training part of common video anomaly detection datasets: UCSD PED1 & PED2, CHUK Avenue, and ShanghaiTech Campus. The proposed framework is evaluated on video sequences with unseen scene contexts in the UCSD PED2 and ShanghaiTech Campus datasets. The detection accuracy and efficiency are also evaluated in detail, and the proposed method for abnormal event detection achieves the highest AUC performance, 84.6%, on the ShanghaiTech Campus dataset and relatively high AUC performance, 96.9% and 74.8%, on UCSD PED2 & PED1 datasets. Compared with other state-of-the-art works, the performance analysis and results confirm the robustness and effectiveness of our proposed framework for cross-scene abnormal event detection.

Seismic assessment of glulam frames with dual-tube self-centering buckling-restrained braces

The development of resilient lateral load-resisting systems is essential for multi-story and high-rise timber buildings in earthquake-prone areas. In this paper, an attempt is made to integrate dual-tube self-centering buckling-restrained braces (DT-SCBRBs) to glulam frames, aiming to improve their structural resilience to earthquakes. To assess the seismic effectiveness of such a DT-SCBRB glulam frame, nonlinear time-history analyses (NLTHAs) were conducted on a series of prototype DT-SCBRB glulam frames with different building heights and design parameters of DT-SCBRBs. The seismic performance of these prototype structures was evaluated in terms of maximum inter-story drift ratios (MaxISDRs), residual inter-story drift ratios (ResISDRs), and inter-story drift uniformity. The results show that the frame with lower post-tensioning-to-yielding ratios of DT-SCBRBs tended to exhibit lower MaxISDRs but could sometimes lead to higher ResISDRs under major earthquakes. Low deformability of tendons could result in tendon fracture under major earthquakes. The ResISDRs of all the prototype frames were lower than 0.5 %, the drift limit for the “Repairable” performance level. Compared to minor and moderate earthquakes, the DT-SCBRB glulam frames deformed more uniformly during major earthquakes. Incremental dynamic analyses (IDAs) were also conducted on these prototype structures, developing a database to quantify performance levels of DT-SCBRB glulam frames. The MaxISDRs of 0.5 %, 0.7 %, and 2.6 % were recommended for immediate occupancy (IO), life safety (LS), and collapse prevention (CP) limit states of DT-SCBRB glulam frames.

Delving into CLIP latent space for Video Anomaly Recognition

We tackle the complex problem of detecting and recognising anomalies in surveillance videos at the frame level, utilising only video-level supervision. We introduce the novel method AnomalyCLIP, the first to combine Vision and Language Models (VLMs), such as CLIP, with multiple instance learning for joint video anomaly detection and classification. Our approach specifically involves manipulating the latent CLIP feature space to identify the normal event subspace, which in turn allows us to effectively learn text-driven directions for abnormal events. When anomalous frames are projected onto these directions, they exhibit a large feature magnitude if they belong to a particular class. We also leverage a computationally efficient Transformer architecture to model short- and long-term temporal dependencies between frames, ultimately producing the final anomaly score and class prediction probabilities. We compare AnomalyCLIP against state-of-the-art methods considering three major anomaly detection benchmarks, i.e. ShanghaiTech, UCF-Crime, and XD-Violence, and empirically show that it outperforms baselines in recognising video anomalies. Project website and code are available at https://lucazanella.github.io/AnomalyCLIP/.

Development and testing of a deep learning algorithm to detect lung consolidation among children with pneumonia using hand-held ultrasound.

Severe pneumonia is the leading cause of death among young children worldwide, disproportionately impacting children who lack access to advanced diagnostic imaging. Here our objectives were to develop and test the accuracy of an artificial intelligence algorithm for detecting features of pulmonary consolidation on point-of-care lung ultrasounds among hospitalized children. This was a prospective, multicenter center study conducted at academic Emergency Department and Pediatric inpatient or intensive care units between 2018-2020. Pediatric participants from 18 months to 17 years old with suspicion of lower respiratory tract infection were enrolled. Bedside lung ultrasounds were performed using a Philips handheld Lumify C5-2 transducer and standardized protocol to collect video loops from twelve lung zones, and lung features at both the video and frame levels annotated. Data from both affected and unaffected lung fields were split at the participant level into training, tuning, and holdout sets used to train, tune hyperparameters, and test an algorithm for detection of consolidation features. Data collected from adults with lower respiratory tract disease were added to enrich the training set. Algorithm performance at the video level to detect consolidation on lung ultrasound was determined using reference standard diagnosis of positive or negative pneumonia derived from clinical data. Data from 107 pediatric participants yielded 117 unique exams and contributed 604 positive and 589 negative videos for consolidation that were utilized for the algorithm development process. Overall accuracy for the model for identification and localization of consolidation was 88.5%, with sensitivity 88%, specificity 89%, positive predictive value 89%, and negative predictive value 87%. Our algorithm demonstrated high accuracy for identification of consolidation features on pediatric chest ultrasound in children with pneumonia. Automated diagnostic support on an ultraportable point-of-care device has important implications for global health, particularly in austere settings.

Learning a multi-cluster memory prototype for unsupervised video anomaly detection

In recent years, there has been rapid development in video anomaly detection (VAD). The previous methods ignored the differences between normal videos and only emphasized learning the commonalities of normal videos. In order to improve the performance of anomaly detection, we delve into the spatial distribution of normal video features and utilize their differences for clustering, leading to more minor reconstruction errors for normal videos and more significant reconstruction errors for abnormal videos. To achieve this goal, we introduce a Multi-Cluster Memory Prototype framework (MCMP) for VAD, which explores the coarse-grained and fine-grained information from video snippet features simultaneously to learn a memory prototype, thereby significantly improving the ability to discriminate abnormal events in complex scenes. First, a video feature clustering method that employs contrastive learning is introduced to group samples sharing similar fine-grained features. Second, the memory mechanism is used to capture the feature distribution of normal samples. Lastly, the Gaussian filter feature transformation method is introduced to make normal and abnormal features more distinguishable. The frame level AUC of MCMP on ShanghaiTech and UCF-Crime benchmark datasets has increased by 1.26% and 0.45% compared to state-of-the-art methods. Our code is publicly available athttps://github.com/WuIkun5658/MCMP.

Expression Patterns of MOTS-c in Adrenal Tumors: Results from a Preliminary Study.

Adrenal tumors, such as adrenocortical carcinoma (ACC), adrenocortical adenoma (ACA), and pheochromocytoma (PCC) are complex diseases with unclear causes and treatments. Mitochondria and mitochondrial-derived peptides (MDPs) are crucial for cancer cell survival. The primary aim of this study was to analyze samples from different adrenal diseases, adrenocortical carcinoma, adrenocortical adenoma, and pheochromocytoma, and compare them with normal adrenal tissue to determine whether the expression levels of the mitochondrial open reading frame of the 12S rRNA type-c (MOTS-c) gene and protein vary between different types of adrenal tumors compared to healthy controls using qPCR, ELISA, and IHC methods. Results showed decreased MOTS-c mRNA expression in all adrenal tumors compared to controls, while serum MOTS-c protein levels increased in ACA and PCC but not in ACC. The local distribution of MOTS-c protein in adrenal tissue was reduced in all tumors. Notably, MOTS-c protein expression declined with ACC progression (stages III and IV) but was unrelated to patient age or sex. Tumor size and testosterone levels positively correlated with MOTS-c mRNA but negatively with serum MOTS-c protein. Additionally, serum MOTS-c protein correlated positively with glucose, total cholesterol, HDL, LDL, and SHGB levels. These findings suggest disrupted expression of MOTS-c in the spectrum of adrenal diseases, which might be caused by mechanisms involving increased mitochondrial dysfunction and structural changes in the tissue associated with disease progression. This study provides a detailed examination of MOTS-c mRNA and protein in adrenal tumors, indicating the potential role of MDPs in tumor biology and progression.

Spatiotemporal Decouple-and-Squeeze Contrastive Learning for Semisupervised Skeleton-Based Action Recognition.

Contrastive learning has been successfully leveraged to learn action representations for addressing the problem of semisupervised skeleton-based action recognition. However, most contrastive learning-based methods only contrast global features mixing spatiotemporal information, which confuses the spatial- and temporal-specific information reflecting different semantic at the frame level and joint level. Thus, we propose a novel spatiotemporal decouple-and-squeeze contrastive learning (SDS-CL) framework to comprehensively learn more abundant representations of skeleton-based actions by jointly contrasting spatial-squeezing features, temporal-squeezing features, and global features. In SDS-CL, we design a new spatiotemporal-decoupling intra-inter attention (SIIA) mechanism to obtain the spatiotemporal-decoupling attentive features for capturing spatiotemporal specific information by calculating spatial- and temporal-decoupling intra-attention maps among joint/motion features, as well as spatial- and temporal-decoupling inter-attention maps between joint and motion features. Moreover, we present a new spatial-squeezing temporal-contrasting loss (STL), a new temporal-squeezing spatial-contrasting loss (TSL), and the global-contrasting loss (GL) to contrast the spatial-squeezing joint and motion features at the frame level, temporal-squeezing joint and motion features at the joint level, as well as global joint and motion features at the skeleton level. Extensive experimental results on four public datasets show that the proposed SDS-CL achieves performance gains compared with other competitive methods.

Self-supervised spatial–temporal feature enhancement for one-shot video object detection

One-shot video object detection is a task that aims to locate and identify objects in video sequences given only a single video sample for each class. Exploration in this field is still in its infancy and previous few-shot video object detection methods have limitation in this task. In this paper, we propose the Self-supervised Feature Enhancement (SFE) framework to address one-shot video object detection task. SFE includes two important modules: Hybrid Spatial Self-supervised Feature Enhancement (HSSFE) and Dynamic Temporal Self-supervised Feature Enhancement (DTSFE). HSSFE enhances features from a spatial perspective with spatial self-supervised auxiliary tasks at frame and instance levels. DTSFE on the other hand enhances features from a temporal perspective with memory-based self-supervised constraint for the same object across different frames. We have conducted experiments on multiple benchmarks, and the results demonstrate that our method achieves state-of-the-art performance.

Guiding tourists’ cautious behaviour: the interaction role of goal message framing and construal level message during the public health crisis context

ABSTRACT Crisis communication is crucial for tourists’ safety, few studies have explored the impact of goal message framing and construal level message on tourists’ cautious behaviours in the public health crisis context. This study investigated the interaction effects of goal message framing effects (gain vs. loss) and construal level message (concrete vs. abstract) on tourists’ risk perception and cautious behaviours using two experiments. The findings suggest that the combination of a loss message and a concrete message, as well as the combination of a gain message and an abstract message, can effectively guide tourists’ cautious behaviours. Among these, the amalgamation of loss message framing with a concrete message is more effective and significant. Simultaneously, the persuasive effect of goal message framing and construal level on tourists’ cautious behaviours is mediated through risk perception. The results of this study can have better suggestions and responses for government units and tourism organisations in tourism crisis management.

When Half Is at Least 50%: Effect of “Framing” and Probability Level on Frequency Estimates

ABSTRACTExpert judgment often involves estimating magnitudes, such as the frequency of deaths due to a pandemic. Three experiments (Ns = 902, 431, and 755, respectively) were conducted to examine the effect of outcome framing (e.g., half of a threatened group expected to survive vs. die), probability level (low vs. high), and probability format (verbal, numeric, or combined) on the estimated frequency of survivals/deaths. Each experiment found an interactive effect of frame and probability level, which supported the hypothesis that forecasted outcomes received by participants were implicitly quantified as lower bounds (i.e., “at least half”). Responding in a manner consistent with a lower‐bound “at least” interpretation was unrelated to incoherence (Experiments 1 and 2) and positively related to numeracy (Experiments 1 and 3), verbal reasoning (Experiment 3), and actively open‐minded thinking (Experiments 2 and 3). The correlational results indicate that implicit lower bounding is an aspect of linguistic inference and not a cognitive error. Implications for research on framing effects are discussed.

Communicating Green Innovation to Online Communities: Evidence from Sports Mega Events

Tourism and event organizers implement green innovation to mitigate the negative environmental impacts they cause. However, how to effectively communicate these green innovations to online communities remains a challenge. Drawing on message framing and construal level theories, we collected 503 tweets and used an exploratory sequential mixed-methods approach to analyze the textual data. First, we conducted abductive coding analysis, identifying two novel framing forms, innovative and value chain framings. Second, using negative binomial regressions, we found that input-value (vs. output-value) framing is often used but decreases online engagement. Moreover, proximal (vs. distal) framing is less used but enhances online engagement. Third, we explored the moderating effect of green innovation, revealing that innovative and proximal framings are more effective for green organizational innovation, whereas emotional framing is more effective for green service innovation. This study offers novel insights on how to strategically communicate green innovation to event online communities.

VPE-WSVAD: Visual prompt exemplars for weakly-supervised video anomaly detection

Weakly Supervised Video Anomaly Detection (WSVAD) plays a crucial role in visual surveillance by effectively distinguishing anomalies from normality with only video-level annotations; nevertheless, due to inherent limitations, including imbalanced data, intra-bag similarity, and snippet entanglement, existing methodologies prone to a bias towards normality. This paper proposes a novel framework, Visual Prompt Exemplars-based WSVAD termed VPE-WSVAD, to enhance discriminative representations of anomalies against normality by incorporating scenario-awareness visual prompt exemplars. The proposed VPE-WSVAD framework comprises three key components. Firstly, the prompt retrieval network generates potential abnormal proposals consistent with visual prompt exemplars at the frame level. Secondly, the proposal filtering mechanism aims to select the proposal with the highest confidence score to represent potential anomalies in the snippet. Finally, prompt likelihood learning is designed to capture the correlations between proposals (w.r.t prompt exemplars) and snippets, generating discriminative representations for each snippet. By incorporating visual prompt exemplars, our method provides more detailed reporting of abnormal events, including when, where, and what. To validate the efficacy of our method, we conducted comprehensive evaluations on three publicly available datasets. The experimental results conclusively demonstrate the superiority of our approach, achieving an impressive frame-level area under the curve (AUC) of 96.88% on the ShanghaiTech dataset and 99.86% on the UCSD Ped2 dataset. Our code will be released in the future.

Irregular Event Classification in Surveillance Videos Using an Auto-Encoder Deep Learning Framework

Anomaly detection is a major deep-learning problem that has a vast area to work upon. We have used a Convolutional Long Short-Term Memory (LSTM) Auto-encoder for the detection of irregular events based on regular events. Anomaly detection can be approached as a supervised problem, but the problem arises when we dig into varieties of anomalies which has many options and can be difficult to supervise, so the use of an unsupervised method is taken, and the results are calculated based on the regularity score of the sequences. The higher the score of regularity, the higher the chances of regularity in the sequence of video frames. The anomaly is detected on the frame level i.e. which frame has the irregularity, and which has regularity in it. The proposed model outperforms in terms of precision (0.97), recall (0.91), and f-measure (0.94) than the existing techniques.

Motion history images: a new method for tracking microswimmers in 3D

Quantitative tracking of rapidly moving micron-scale objects remains an elusive challenge in microscopy due to low signal-to-noise. This paper describes a novel method for tracking micron-sized motile organisms in off-axis Digital Holographic Microscope (DHM) raw holograms and/or reconstructions. We begin by processing the microscopic images with the previously reported Holographic Examination for Life-like Motility (HELM) software, which provides a variety of tracking outputs including motion history images (MHIs). MHIs are stills of videos where the frame-to-frame changes are indicated with color time-coding. This exposes tracks of objects that are difficult to identify in individual frames at a low signal-to-noise ratio. The visible tracks in the MHIs are superior to tracks identified by all tested automated tracking algorithms that start from object identification at the frame level, particularly in low signal-to-noise ratio data, but do not provide quantitative track data. In contrast to other tracking methods, like Kalman filter, where the recording is analyzed frame by frame, MHIs show the whole time span of particle movement at once and eliminate the need to identify objects in individual frames. This feature also enables post-tracking identification of low-SNR objects. We use these tracks, rather than object identification in individual frames, as a basis for quantitative tracking of Bacillus subtilis by first generating MHIs from X, Y, and t stacks (raw holograms or a projection over reconstructed planes), then using a region-tracking algorithm to identify and separate swimming pathways. Subsequently, we identify each object's Z plane of best focus at the corresponding X, Y, and t points, yielding ap full description of the swimming pathways in three spatial dimensions plus time. This approach offers an alternative to object-based tracking for processing large, low signal-to-noise datasets containing highly motile organisms.

Video Moment Retrieval With Noisy Labels.

Video moment retrieval (VMR) aims to localize the target moment in an untrimmed video according to the given nature language query. The existing algorithms typically rely on clean annotations to train their models. However, making annotations by human labors may introduce much noise. Thus, the video moment retrieval models will not be well trained in practice. In this article, we present a simple yet effective video moment retrieval framework via bottom-up schema, which is in end-to-end manners and robust to noisy label training. Specifically, we extract the multimodal features by syntactic graph convolutional networks and multihead attention layers, which are fused by the cross gates and the bilinear approach. Then, the feature pyramid networks are constructed to encode plentiful scene relationships and capture high semantics. Furthermore, to mitigate the effects of noisy annotations, we devise the multilevel losses characterized by two levels: a frame-level loss that improves noise tolerance and an instance-level loss that reduces adverse effects of negative instances. For the frame level, we adopt the Gaussian smoothing to regard noisy labels as soft labels through the partial fitting. For the instance level, we exploit a pair of structurally identical models to let them teach each other during iterations. This leads to our proposed robust video moment retrieval model, which experimentally and significantly outperforms the state-of-the-art approaches on standard public datasets ActivityCaption and textually annotated cooking scene (TACoS). We also evaluate the proposed approach on the different manual annotation noises to further demonstrate the effectiveness of our model.

Discerning Deception: A Face-Centric Deepfake Detection Approach with ResNeXt-50 and LSTMs

Abstract: A.I. has grown to epidemic proportions over the last years as its applied in almost all sectors to allocate workload from humans but end up being done effectively with no human intervention. A branch of A.I. called deep learning, which operates by mimicking human judgment and action through neural network systems. Nonetheless, with the increase height of the two platforms have been experienced sufficient cases of misguided individuals using tools to recycle videos, audios, and texts to achieve their agendas. This insinuates a due assumption that Generative Adversarial Networks, GANs, are central to the development of believable deepfakes. GANs have developed a crucial ability to generate videos that replace frames with material from another video source to create deepfakes videos. While GANs serves various purposes such as entertainment, teaching, and experimentation, malicious actors can misuse these deep learning techniques to manipulate videos, impacting the privacy of individuals in society. This paper conducts an analysis of different deepfake detection models, comparing their efficacy and discussing potential future extensions of deepfake technology. The study presents a novel deepfake detection approach utilizing a combination of Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN). This method utilizes ResNext50 for extracting features at the frame level, while employing LSTM (Long Short-Term Memory) for video classification based on these extracted features. Various datasets are incorporated, including the deepfake detection challenge dataset (DFDC) and Face Forensics deepfake collections (FF++), combining them to achieve a high-accuracy model capable of accurately discerning between real and deepfake videos. The results of this study make a valuable contribution to the continuous endeavors aimed at improving deepfake detection abilities and ensuring privacy protection in a time heavily influenced by artificial intelligence

A Deep Approach to Deep Fake Detection

In recent months, the proliferation of free deep learning-based software tools has facilitated the creation of credible face exchanges in videos, resulting in what are known as "DeepFake" (DF) videos. While manipulations of digital videos have been demonstrated for several decades through the use of visual effects, recent advances in deep learning have significantly increased the realism of fake content and the accessibility with which it can be created. These AI-synthesized media, popularly referred to as DF, pose a significant challenge for detection. Detecting DF is a major challenge due to the complexity of training algorithms to spot them. In this work, we propose a detection approach using Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). Our system leverages a CNN to extract features at the frame level, which are then used to train an RNN. The RNN learns to classify whether a video has been manipulated, detecting temporal inconsistencies between frames introduced by DF creation tools. We evaluate our approach against a large set of fake videos collected from standard datasets and demonstrate competitive results using a simple architecture.

Deep-Learning-Based Segmentation of Predefined Chunks in Connected Speech: A Retrospective Analysis

Background and Objectives In institutional settings, manually segmenting connected speech is a time-consuming and labor-intensive process. This study aims to develop a deep-learning model for automating this process, evaluating its accuracy, and determining the minimum dataset size for effective performance.Materials and Method Voice data from 524 individuals with pathological conditions and 502 individuals with normal conditions, totaling 1026 samples, were used. Each voice sample had 17 chunks, including a “summer” sentence (15 chunks) and vowels /α/ and /i/. The deep-learning model employed in this study is based on the multi-layer perceptron-mixer architecture. This study evaluated performance using the Intersection over Union (IoU) metric, commonly employed in artificial intelligence-based image detection for chunk segmentation.Results The accuracy of chunk identification at the frame level was 96.47%. Using IoU metrics, chunk segmentation accuracy was 98.15% at IoU ≥0.6, 96.03% at IoU ≥0.7, and 89.78% at IoU ≥0.8. Optimal dataset size exploration indicated that more than 700 connected speech datasets were needed for successful training, maintaining F1-scores up to 95% at IoU ≥0.7.Conclusion The artificial intelligence model is suitable for the development of an automated system that efficiently divides segments in the institutional collection of voice data. This suggests its potential utility in advancing voice research using connected speech.

Exposing Fake Faces Through Deep Neural Networks Combining Content and Trace Feature Extractors

Abstract: In recent times, the proliferation of free deep learning-based software has facilitated the emergence of convincing facial swaps in videos, commonly referred to as ‘DeepFake’ (DF) videos. ‘Deep learning’ has improved the realism and accessibility of creating fake digital video content, which was previously attainable through traditional visual effects. These AIgenerated media, often referred to as DF, present a dual challenge: their creation is relatively straightforward using AI tools, yet their detection poses a significant hurdle. We address this challenge by employing Convolutional Neural Networks (CNNs) and ‘Recurrent Neural Networks’ (RNNs) to identify ‘DFs’. Specifically, our system utilizes a CNN to obtain frame level characteristics and apply them to train an RNN capable of identifying temporal inconsistencies introduced by DF creation tools. We evaluate our approach on a substantial dataset of fake videos and demonstrate competitive performance with a straightforward architecture.

Automatic Analysis and Event Detection Technology of Sports Competition Video Based on Deep Learning

This paper presents an investigation into the application of advanced techniques, including deep learning classification and FCM centroid segmentation, for automated event detection and analysis in sports videos. Through a series of experiments and analyses, we demonstrate the effectiveness of deep learning models in accurately categorizing events within sports footage, alongside the insights provided by FCM segmentation into event occurrences at the frame level. This paper investigates the application of advanced techniques, such as deep learning classification and FCM centroid segmentation, for automated event detection in sports videos. Through experiments, we achieved an average accuracy of 93.2% using deep learning classification and identified key events with probabilities ranging from 0.01 to 0.90 using FCM segmentation.