Video Frames Research Articles

Wi-Filter: WiFi-Assisted Frame Filtering on the Edge for Scalable and Resource-Efficient Video Analytics

With the growing prevalence of large-scale intelligent surveillance camera systems, the burden on real-time video analytics pipelines has significantly increased due to continuous video transmission from numerous cameras. To mitigate this strain, recent approaches focus on filtering irrelevant video frames early in the pipeline, at the camera or edge device level. In this paper, we propose Wi-Filter, an innovative filtering method that leverages Wi-Fi signals from wireless edge devices, such as Wi-Fi-enabled cameras, to optimize filtering decisions dynamically. Wi-Filter utilizes channel state information (CSI) readily available from these wireless cameras to detect human motion within the field of view, adjusting the filtering threshold accordingly. The motion-sensing models in Wi-Filter (Wi-Fi assisted Filter) are trained using a self-supervised approach, where CSI data are automatically annotated via synchronized camera feeds. We demonstrate the effectiveness of Wi-Filter through real-world experiments and prototype implementation. Wi-Filter achieves motion detection accuracy exceeding 97.2% and reduces false positive rates by up to 60% while maintaining a high detection rate, even in challenging environments, showing its potential to enhance the efficiency of video analytics pipelines.

A novel hybrid architecture for video frame prediction: combining convolutional LSTM and 3D CNN

A novel hybrid architecture for video frame prediction: combining convolutional LSTM and 3D CNN

Hybrid Neural Network Models to Estimate Vital Signs from Facial Videos

Introduction: Remote health monitoring plays a crucial role in telehealth services and the effective management of patients, which can be enhanced by vital sign prediction from facial videos. Facial videos are easily captured through various imaging devices like phone cameras, webcams, or surveillance systems. Methods: This study introduces a hybrid deep learning model aimed at estimating heart rate (HR), blood oxygen saturation level (SpO2), and blood pressure (BP) from facial videos. The hybrid model integrates convolutional neural network (CNN), convolutional long short-term memory (convLSTM), and video vision transformer (ViViT) architectures to ensure comprehensive analysis. Given the temporal variability of HR and BP, emphasis is placed on temporal resolution during feature extraction. The CNN processes video frames one by one while convLSTM and ViViT handle sequences of frames. These high-resolution temporal features are fused to predict HR, BP, and SpO2, capturing their dynamic variations effectively. Results: The dataset encompasses 891 subjects of diverse races and ages, and preprocessing includes facial detection and data normalization. Experimental results demonstrate high accuracies in predicting HR, SpO2, and BP using the proposed hybrid models. Discussion: Facial images can be easily captured using smartphones, which offers an economical and convenient solution for vital sign monitoring, particularly beneficial for elderly individuals or during outbreaks of contagious diseases like COVID-19. The proposed models were only validated on one dataset. However, the dataset (size, representation, diversity, balance, and processing) plays an important role in any data-driven models including ours. Conclusions: Through experiments, we observed the hybrid model’s efficacy in predicting vital signs such as HR, SpO2, SBP, and DBP, along with demographic variables like sex and age. There is potential for extending the hybrid model to estimate additional vital signs such as body temperature and respiration rate.

Deep Learning-Based Assessment of Lip Symmetry for Patients With Repaired Cleft Lip.

Post-surgical lip symmetry assessment is a key indicator of cleft repair success. Traditional methods rely on distances between anatomical landmarks, which are impractical for video analysis and overlook texture and appearance. We propose an artificial intelligence (AI) approach to automate this process, analyzing lateral lip morphology for a quantitative symmetry evaluation. We utilize contrastive learning to quantify lip symmetry by measuring the similarity between the representations of the sides, which is subsequently used to classify the severity of asymmetry. Our model does not require patient images for training. Instead, we introduce dissimilarities in face images from open datasets using two methods: temporal misalignment for video frames and face transformations to simulate lip asymmetry observed in the target population. The model differentiates the left and right image representations to assess asymmetry. We evaluated our model on 146 images of patients with repaired cleft lip. The deep learning model trained with face transformations categorized patient images into five asymmetry levels, achieving a weighted accuracy of 75% and a Pearson correlation of 0.31 with medical expert human evaluations. The model utilizing temporal misalignment achieved a weighted accuracy of 69% and a Pearson correlation of 0.27 for the same classification task. We propose an automated approach for assessing lip asymmetry in patients with repaired cleft lip by transforming facial images of control subjects to train a deep learning model, eliminating manual anatomical landmarks. Our promising results provide a more efficient and objective tool for evaluating surgical outcomes.

Foreign object debris detection in lane images using deep learning methodology

Background Foreign object debris (FOD) is an unwanted substance that damages vehicular systems, most commonly the wheels of vehicles. In airport runways, these foreign objects can damage the wheels or internal systems of planes, potentially leading to flight crashes. Surveys indicate that FOD-related damage costs over $4 billion annually, affecting airlines, airport tenants, and passengers. Current FOD clearance involves high-cost radars and significant manpower, and existing radar and camera-based surveillance methods are expensive to install. Methods This work proposes a video-based deep learning methodology to address the high cost of radar-based FOD detection. The proposed system consists of two modules for FOD detection: object classification and object localization. The classification module categorizes FOD into specific types of foreign objects. In the object localization module, these classified objects are pinpointed in video frames. Results The proposed system was experimentally tested with a large video dataset and compared with existing methods. The results demonstrated improved accuracy and robustness, allowing the FOD clearance team to quickly detect and remove foreign objects, thereby enhancing the safety and efficiency of airport runway operations.

Blink Detection Using 3D Convolutional Neural Architectures and Analysis of Accumulated Frame Predictions.

Blink detection is considered a useful indicator both for clinical conditions and drowsiness state. In this work, we propose and compare deep learning architectures for the task of detecting blinks in video frame sequences. The first step is the training and application of an eye detector that extracts the eye regions from each video frame. The cropped eye regions are organized as three-dimensional (3D) input with the third dimension spanning time of 300 ms. Two different 3D convolutional neural networks are utilized (a simple 3D CNN and 3D ResNet), as well as a 3D autoencoder combined with a classifier coupled to the latent space. Finally, we propose the usage of a frame prediction accumulator combined with morphological processing and watershed segmentation to detect blinks and determine their start and stop frame in previously unseen videos. The proposed framework was trained on ten (9) different participants and tested on five (8) different ones, with a total of 162,400 frames and 1172 blinks for each eye. The start and end frame of each blink in the dataset has been annotate by specialized ophthalmologist. Quantitative comparison with state-of-the-art blink detection methodologies provide favorable results for the proposed neural architectures coupled with the prediction accumulator, with the 3D ResNet being the best as well as the fastest performer.

Adaptation optimizes sensory encoding for future stimuli.

Sensory neurons continually adapt their response characteristics according to recent stimulus history. However, it is unclear how such a reactive process can benefit the organism. Here, we test the hypothesis that adaptation actually acts proactively in the sense that it optimally adjusts sensory encoding for future stimuli. We first quantified human subjects' ability to discriminate visual orientation under different adaptation conditions. Using an information theoretic analysis, we found that adaptation leads to a reallocation of coding resources such that encoding accuracy peaks at the mean orientation of the adaptor while total coding capacity remains constant. We then asked whether this characteristic change in encoding accuracy is predicted by the temporal statistics of natural visual input. Analyzing the retinal input of freely behaving human subjects showed that the distribution of local visual orientations in the retinal input stream indeed peaks at the mean orientation of the preceding input history (i.e., the adaptor). We further tested our hypothesis by analyzing the internal sensory representations of a recurrent neural network trained to predict the next frame of natural scene videos (PredNet). Simulating our human adaptation experiment with PredNet, we found that the network exhibited the same change in encoding accuracy as observed in human subjects. Taken together, our results suggest that adaptation-induced changes in encoding accuracy prepare the visual system for future stimuli.

Clustering fast optimization strategy for holographic video displays.

Computer-generated holography (CGH) is an advanced technology for three-dimensional (3D) displays. While the stochastic gradient descent (SGD) method is effective for holographic optimization, its application to holographic video displays is computationally expensive, as each frame requires separate optimization. To address this, we propose a novel, to the best of our knowledge, clustering optimization strategy to accelerate the SGD process for holographic video displays. Our method exploits the inherent similarities between video frames by jointly optimizing shared features first, followed by the frame-specific optimization of unique features, thereby minimizing redundant computations. The process involves clustering video frames based on common features and using the optimized holograms of the cluster center image, along with global scale factors, as initial conditions for each frame within the cluster. Numerical simulations and optical experiments demonstrate that the proposed method achieves approximately a twofold increase in computational efficiency, significantly enhancing the feasibility of holographic video displays for broader applications.

Road Accident Emergency Reporting Management System using Neural Network

Given the high number of road accidents that result in deaths every day, victims may lose their lives if emergency medical personnel or concerned parties fail to arrive promptly to intervene in an accident or if they do so, because they are unaware of the location of a nearby hospital or medical facility, they may not be able to provide the much-needed help. To avert this scenario, this research work seeks to deploy a Deep Learning Road Accident Emergencies Management model, which by examining of video and sensors’ data, identifies and reports traffic incidents in real time. A Neural Network (NN) is used by the model to recognize accident scenarios based on anomalous motion patterns inside video frames, including sudden halts and crashes. Put into practice in python, the program incorporates deep learning libraries such as OpenCV and TensorFlow for strong data processing and analysis in real time. To train and evaluate the model, an 80/20 data split, which results in reduced latency and high accuracy in accident detection. The Key performance metrics include a testing accuracy of 96%, an F1-score of 87.5%, and a response time of less than two seconds on average. The study advances the domains of artificial intelligence, transportation safety, and emergency management by offering a practical, real-time solution for automated accident reporting. This quick detection capability allows for timely emergency alerts, which are sent to responders via a web-based interface, giving them precise location details. The model's promise as a life-saving tool in smart city infrastructures is highlighted by its capacity to deliver precise and prompt replies. The Road Accident Emergency Reporting Model has important ramifications for increasing road safety, decreasing emergency response times, and boosting public safety in general due to its strong design and low latency.

Enhancing Underwater Video from Consecutive Frames While Preserving Temporal Consistency

Current methods for underwater image enhancement primarily focus on single-frame processing. While these approaches achieve impressive results for static images, they often fail to maintain temporal coherence across frames in underwater videos, which leads to temporal artifacts and frame flickering. Furthermore, existing enhancement methods struggle to accurately capture features in underwater scenes. This makes it difficult to handle challenges such as uneven lighting and edge blurring in complex underwater environments. To address these issues, this paper presents a dual-branch underwater video enhancement network. The network synthesizes short-range video sequences by learning and inferring optical flow from individual frames. It effectively enhances temporal consistency across video frames through predicted optical flow information, thereby mitigating temporal instability within frame sequences. In addition, to address the limitations of traditional U-Net models in handling complex multiscale feature fusion, this study proposes a novel underwater feature fusion module. By applying both max pooling and average pooling, this module separately extracts local and global features. It utilizes an attention mechanism to adaptively adjust the weights of different regions in the feature map, thereby effectively enhancing key regions within underwater video frames. Experimental results indicate that when compared with the existing underwater image enhancement baseline method and the consistency enhancement baseline method, the proposed model improves the consistency index by 30% and shows a marginal decrease of only 0.6% in enhancement quality index, demonstrating its superiority in underwater video enhancement tasks.

Brainsourcing for temporal visual attention estimation

AbstractThe concept of temporal visual attention in dynamic contents, such as videos, has been much less studied than its spatial counterpart, i.e., visual salience. Yet, temporal visual attention is useful for many downstream tasks, such as video compression and summarisation, or monitoring users’ engagement with visual information. Previous work has considered quantifying a temporal salience score from spatio-temporal user agreements from gaze data. Instead of gaze-based or content-based approaches, we explore to what extent only brain signals can reveal temporal visual attention. We propose methods for (1) computing a temporal visual salience score from salience maps of video frames; (2) quantifying the temporal brain salience score as a cognitive consistency score from the brain signals from multiple observers; and (3) assessing the correlation between both temporal salience scores, and computing its relevance. Two public EEG datasets (DEAP and MAHNOB) are used for experimental validation. Relevant correlations between temporal visual attention and EEG-based inter-subject consistency were found, as compared with a random baseline. In particular, effect sizes, measured with Cohen’s d, ranged from very small to large in one dataset, and from medium to very large in another dataset. Brain consistency among subjects watching videos unveils temporal visual attention cues. This has relevant practical implications for analysing attention for visual design in human-computer interaction, in the medical domain, and in brain-computer interfaces at large.

From video to vital signs: a new method for contactless multichannel seismocardiography

Seismocardiography (SCG) is a technique that non-invasively measures the chest wall’s local vibrations caused by the heart’s mechanical activity. Traditionally, SCG signals have been recorded using accelerometers placed at a single location on the chest wall. This study presents an innovative, cost-effective SCG method that utilizes standard smartphone videos to capture data from multiple chest locations. The analysis of vibrations from multiple points can offer a more thorough understanding of the heart’s mechanical activity compared to signals obtained solely from a single chest location. Our approach employs computer vision and deep learning techniques to extract and improve the resolution of multichannel SCG maps obtained by video capture of chest movement. We attached a grid of patterned stickers to the chest surface and recorded videos of chest movements during different respiratory phases. Using a deep learning-based object detector and a template tracking method, we tracked the stickers across video frames and extracted the corresponding SCG signals from sticker displacements. We also developed a robust algorithm to estimate heart rate (HR) from these chest videos and identify the optimal chest location for HR estimation. The method was tested on 28 chest videos captured from 14 healthy participants. The results demonstrated that our method effectively extracted multichannel SCG maps and enhanced their resolution with a mean squared error of 0.1078 and 0.0418 for right-to-left and head-to-foot SCG signals, respectively. We observed intersubject chest vibration patterns corresponding to cardiac events including opening and closure of the heart valves. Moreover, our algorithm accurately estimated HR from 1968 SCG signals extracted from the videos compared to the gold-standard HR measured from each subject’s electrocardiogram (bias ± 1.96 SD = 0.04 ± 2.14 bpm; r = 0.99, p < 0.001). The findings from this study underscore the potential of our approach in developing a cardiac monitoring tool using a smartphone that would be widely accessible to the general public and might provide more timely detection of diseases.

An effective video captioning based on language description using a novel Graylag Deep Kookaburra Reinforcement Learning

Video captioning exhibits a complex challenge, particularly due to the increased subject intensity within videos compared to image caption generation. The presence of redundant visual information in video data adds complexity for captioners, making it difficult to simplify various content and eliminate irrelevant elements. Additionally, this redundancy often results in misalignment with equivalent visual semantics in the ground truth, further complicating the video captioning process. In response to these challenges, this research introduces the Graylag Deep Kookaburra Reinforcement Learning (GDKRL) framework, which enhances video captioning through a multi-stage process. First, object detection is performed using the single-shot multi-box detector with generalized intersection over union for accurate object tracking and similarity calculation. Next, the gazelle autoencoder extracts and fuses features from video frames, integrating complex visual and temporal information into a unified representation. The residual convolved dual sparse graph attention network then generates detailed and contextually rich language descriptions by applying dual sparse attention mechanisms and residual convolutions. Finally, hybrid graylag and kookaburra optimization refine the captioning process, producing comprehensive and precise textual descriptions of the video content. Extensive experiments on MSVD yielded 81.79 BLEU-4, 51.2 METEOR, 133.3 CIDEr and 81.7 ROUGE-L; VATEX achieved 62.29 BLEU-4, 51.2 METEOR, 110.2 CIDEr and 78.45 ROUGE-L; and the MSR-VTT dataset obtained 44.52 BLEU-4, 33.35 METEOR, 63.9 CIDEr and 68.9 ROUGE-L, demonstrating that the proposed technique significantly outperforms previous approaches and highlights its effectiveness.

Recurrent Flow Update Model Using Image Pyramid Structure for 4K Video Frame Interpolation

Video frame interpolation (VFI) is a task that generates intermediate frames from two consecutive frames. Previous studies have employed two main approaches to extract the necessary information from both frames: pixel-level synthesis and flow-based methods. However, when synthesizing high-resolution videos using VFI, each approach has its limitations. Pixel-level synthesis based on the transformer architecture requires high complexity to achieve 4K video results. In the case of flow-based methods, forward warping can produce holes where pixels are not allocated, while backward warping approaches struggle to obtain accurate backward flow. Additionally, there are challenges during the training stage; previous works have often generated suboptimal results by training multi-stage model architectures separately. To address these issues, we propose a Recurrent Flow Update (RFU) model trained in an end-to-end manner. We introduce a global flow update module that leverages global information to mitigate the weaknesses of forward flow and gradually correct errors. We demonstrate the effectiveness of our method through several ablation studies. Our approach achieves state-of-the-art performance not only on the XTest and Davis datasets, which have 4K resolution, but also on the SNU-FILM dataset, which features large motions at low resolution.

High-Definition, Video-Rate Triple-Channel NIR-II Imaging Using Shadowless Lamp Excitation and Illumination.

Multichannel imaging in the second near-infrared (NIR-II) window offers vital and comprehensive information for complex surgical environments, yet a simple, high-quality, video-rate multichannel imaging method with low safety risk remains to be proposed. Centered at the superior NIR-IIx window of 1400-1500 nm, triple-channel imaging coordinated with 1000-1100 and 1700-1880 nm (NIR-IIc) achieves exceptional clarity and an impressive signal-to-crosstalk ratio as high as 22.10. To further simplify the light source and lower the safety risk, we develop a type of in vivo multichannel imaging-assisted surgical navigation mode at a video frame rate of 25 fps under shadowless lamp excitation and illumination instead of extra excitation light sources. This work provides a reference for real-time, high-imaging-performance multichannel imaging with minimal crosstalk and introduces a practical fluorescence surgical navigation paradigm.

Design and Implementation of Sports Network Teaching Platform Based on Discrete Similarity Method

Sport is the most important topic in education, which individually shows motor skills in health activities. Athletic training should focus graduates on sports, low-involvement task activities, and physical fitness or exercise. Teaching concepts and methodologies are innovative, coaching methods and procedures, evaluation of coaching sessions in sports-all this is accompanied by development in the sports hall and successful improvement of sports performance. Here, he provides extraordinary assistance to students by predicting early school leaving and improving the potential application of wireless platforms in sports programs and changing the nature of sports, including visualization and repetition, and incorporating it into sports training. A sports network learning platform based on the discrete similarity approach is proposed for design and implementation in this paper. First, data is collected from a real-time data set. After that, the data submitted for preprocessing removes noise and imperfect records, which can be removed using a global constant or a most likely learning method using an Adaptive Savitsky-Golay Filtering (ASGF) method. The preprocessing method is fed into the feature extraction utilizing Multi‐Hypothesis Fuzzy‐Matching Radon Transform (MHFMRT). MHFMRT extract statistical features such us kurtosis, variance, energy, mean and standard deviation. Then, the Similarity-Based Convolutional Neural Network (SBCNN) method is optimized by the Chaotic Satin Bower Bird Optimization (CSBBO) algorithm, which effectively classifies the visualization and a satisfactory training platform of the sports network. The proposed system is executed in the MATrix LABoratory (MATLAB) platform and the performance of the proposed methods attains 28.5%, 27.9% and 29.5% higher accuracy, 28%, 21% and 26.5% higher precision, 21.2%, 28.75% and 21.3% higher recall compared with the existing methods like Recurrent Neural model to analyze the effect of Physical Training and Treatment in Relation to Sports Injuries (PTT-RSI-RNN), recognizing Sports Activities from Video Frames using Deformable Convolution and Adaptive Multiscale Features (SAVF-DCN) and Sports Match Prediction model for training and Exercise using attention-based LSTM network (SMPE-LSTM) respectively

Quantum Machine Learning for Video Compression: An Optimal Video Frames Compression Model using Qutrits Quantum Genetic Algorithm for Video multicast over the Internet

The transmission of video is greatly aided by video compression. Redundancy (spatial, temporal, statistical, and psycho-visual) within and between video frames is something that video compression approaches aim to get rid of. The degree to which similarity-based redundancy exists between consecutive frames, however, is a function of how often the frames are sampled and how the objects in the scene are moving. Existing neural network-based video compression approaches rely on a static codebook to perform compression, which prevents them from adapting to new video’s data. In order to create an optimal codebook for vector quantization, which is then employed as an activation function inside a neural network's hidden layer, this research offers a modified video compression method based on a Qutrits based Quantum Genetic Algorithm (QQGA). Using quantum parallelization and entanglement of the quantum state, QQGA is capable of solving the same set of problems as a traditional genetic algorithm while considerably accelerating the evolutionary process. The technique is built on the concept of utilizing Qutrits (three-level quantum system) to represent population individuals. The evolution operator, which is responsible for the updates to the quantum system state, has been constructed using a straightforward approach that does not need a lookup table. Compared to qubit, qudit provides a larger state space to store and process information, and thus can enhance the algorithm’s efficiency. To create the context-based initial codebook, the background subtraction algorithm is used to extract moving objects from frames. Moreover, important wavelet coefficients are compressed losslessly using Differential Pulse Code Modulation (DPCM), whereas low energy coefficients are compressed lossy using Learning Vector Quantization neural networks (LVQ). To obtain a high compression ratio, Run-Length Encoding is then used to encode the quantized coefficients. In comparison to the conventional evolutionary algorithm-based video compression method, experiments have shown that the quantum-inspired system may achieve a greater compression ratio with acceptable efficiency as evaluated by PSNR.

SSIM over MSE: A new perspective for video anomaly detection.

Video anomaly detection plays a crucial role in ensuring public safety. Its goal is to detect abnormal patterns contained in video frames. Most existing models distinguish the anomalies based on the Mean Squared Error (MSE), which is hard to align with human perception, resulting in discrepancies between model-detected anomalies and those recognized by humans. Unlike the Human Visual System (HVS), those models are trained to prioritize texture over shape, which leads to poor model interpretability and limited performance. To address these limitations, we propose to optimize the video anomaly detection models from the perspective of human visual relevance. The optimization infrastructure includes a novel Structural Similarity Index (SSIM) based loss, a novel anomaly score calculation method based on SSIM, and a spatial-temporal enhancement block in 3D convolution (STE-3D). SSIM loss helps the model emphasize shape information in videos rather than texture. An anomaly score method based on SSIM evaluates video frames to align more closely with human visual perception. STE-3D improves the model's capacity to capture spatial-temporal features and compensates for the deficiency of the SSIM loss in capturing temporal features. STE-3D is lightweight in design and seamlessly integrated into existing video anomaly detection models based on 3D convolution. Extensive experiments and ablation studies were conducted in four challenging video anomaly detection benchmarks,i.e., UCSD Ped1, UCSD Ped2, CUHK Avenue, and ShanghaiTech. The experimental results validate the efficacy of the proposed approaches in improving video anomaly detection performance.

Human Behaviour Recognition in Video Surveillance Systems Based on Neural Networks

Abstract: "Human behaviour recognition plays a critical role in intelligent video surveillance systems for security, crowd management, and abnormal activity detection. This paper explores the application of neural networks, particularly Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), for recognizing and classifying human behaviours in video surveillance footage. We propose a novel deep learning-based approach to extract features from video frames and temporal data, achieving high accuracy in behaviour recognition tasks. Experimental results on benchmark datasets demonstrate the effectiveness of our model in detecting various behaviours, including walking, running, and suspicious activities, even in complex environments."

Object Tracking Using HSV Values and OpenCV

This research shows how to use colour and movement to automate the process of recognising and tracking things. Video tracking is a technique for detecting a moving object over a long distance using a camera. The main purpose of video tracking is to connect target objects in subsequent video frames. The connection may be particularly troublesome when things move faster than the frame rate. Using HSV colour space values and OpenCV in different video frames, this study proposes a way to track moving objects in real-time. We begin by calculating the HSV value of an item to be monitored, and then we track the object throughout the testing step. The items were shown to be tracked with 90 percent accuracy.