Frames Of Video Sequence Research Articles

Brain-like border ownership signals support prediction of natural videos.

To make sense of visual scenes, the brain must segment foreground from background. This is thought to be facilitated by neurons in the primate visual system that encode border ownership (BOS), i.e. whether a local border is part of an object on one or the other side of the border. It is unclear how these signals emerge in neural networks without a teaching signal of what is foreground and background. In this study, we investigated whether BOS signals exist in PredNet, a self-supervised artificial neural network trained to predict the next image frame of natural video sequences. We found that a significant number of units in PredNet are selective for BOS. Moreover these units share several other properties with the BOS neurons in the brain, including robustness to scene variations that constitute common object transformations in natural videos, and hysteresis of BOS signals. Finally, we performed ablation experiments and found that BOS units contribute more to prediction than non-BOS units for videos with moving objects. Our findings indicate that BOS units are especially useful to predict future input in natural videos, even when networks are not required to segment foreground from background. This suggests that BOS neurons in the brain might be the result of evolutionary or developmental pressure to predict future input in natural, complex dynamic visual environments.

Deep learning for detection and counting of Nephrops norvegicus from underwater videos

Abstract The Norway lobster (Nephrops norvegicus) is one of the most important fishery items for the EU blue economy. This paper describes a software architecture based on neural networks, designed to identify the presence of N. norvegicus and estimate the number of its individuals per square meter (i.e. stock density) in deep-sea (350–380 m depth) Fishery No-Take Zones of the northwestern Mediterranean. Inferencing models were obtained by training open-source networks with images obtained from frames partitioning of in submarine vehicle videos. Animal detections were also tracked in successive frames of video sequences to avoid biases in individual recounting, offering significant success and precision in detection and density estimations.

Multimodal risk classifier for cardiorespiratory diseases taking into account concomitant diseases and synergy effect

The purpose of the research is to develop a methodology for classifying complexly structured halftone images based on a multimodal approach using methods of morphological analysis, spectral analysis and neural network modeling.Methods. A method for classifying the contours of the boundaries of segments of a complexly structured image is described. The method is based on the fact that in chronic diseases of the pancreas, there is a violation of the integrity of the contour of its border and its waviness increases due to retractions and bulges caused by an alterative inflammatory process. The method includes the stages of normalization of ultrasound images and image segmentation with the selection of the contour of the object of interest. To classify the contour of a segment boundary, it is proposed to use Fourier analysis and neural network technologies. The method is illustrated using the example of classifying the contour of the border of the pancreas on its transcutaneous acoustic image.Results. Experimental studies of the proposed methods and means for classifying medical risk were carried out on diagnostic tasks according to the following classes: "chronic pancreatitis" – "without pathology". For experimental studies, video sequences of ultrasound images of the pancreas provided by an endoscopist were used. The purpose of the experimental studies was to analyze the classification quality indicators of image classifiers with class segments "Chronic pancreatitis" and "Without pathology". The training sample of video images (frames of video sequences) included 200 examples, one hundred from each class. The quality indicator "Sensitivity" of classification for two classes is 85,7%, the indicator "Specificity" is 87,1%.Conclusion. The use of the contour analysis method in classifiers of ultrasound images of the pancreas opens up new opportunities for accessible and objective diagnosis of pancreatic diseases, expanding the capabilities of intelligent clinical decision support systems.

Irregular Event Classification in Surveillance Videos Using an Auto-Encoder Deep Learning Framework

Anomaly detection is a major deep-learning problem that has a vast area to work upon. We have used a Convolutional Long Short-Term Memory (LSTM) Auto-encoder for the detection of irregular events based on regular events. Anomaly detection can be approached as a supervised problem, but the problem arises when we dig into varieties of anomalies which has many options and can be difficult to supervise, so the use of an unsupervised method is taken, and the results are calculated based on the regularity score of the sequences. The higher the score of regularity, the higher the chances of regularity in the sequence of video frames. The anomaly is detected on the frame level i.e. which frame has the irregularity, and which has regularity in it. The proposed model outperforms in terms of precision (0.97), recall (0.91), and f-measure (0.94) than the existing techniques.

LipNet: End-to-End Lipreading

Lipreading is the task of decoding text from the movement of a speaker’s mouth. This research presents the development of an advanced end-to-end lipreading system. Leveraging deep learning architectures and multimodal fusion techniques, the proposed system interprets spoken language solely from visual cues, such as lip movements. Through meticulous data collection, annotation, preprocessing, model development, and evaluation, diverse datasets encompassing various speakers, accents, languages, and environmental conditions are curated to ensure robustness and generalization. Conventional methods divided the task into two phases: prediction and designing or learning visual characteristics. Most deep lipreading methods are trainable from end to end. In the past, lipreading has been tackled using tedious and sometimes unsatisfactory techniques that break down speech into smaller units like phonemes or visemes. But these methods often fail when faced with real-world problems, such contextual factors, accents, and differences in speech patterns. Nevertheless, current research on end-to-end trained models only carries out word classification; sentence-level sequence prediction is not included. LipNet is an end-to-end trained model that uses spatiotemporal convolutions, a recurrent network, and the connectionist temporal classification loss to translate a variable-length sequence of video frames to text. LipNet breaks from this traditional paradigm by using an all-encompassing, end-to-end approach supported by deep learning algorithms, Convolutional neural networks (CNNs) and recurrent neural networks (RNNs), which are skilled at processing sequential data and extracting high-level representations, are fundamental to LipNet's architecture.LipNet achieves 95.2% accuracy in sentence-level on the GRID corpus, overlapped speaker split task, outperforming experienced human lipreaders and the previous 86.4% word-level state-of-the-art accuracy.The results underscore the transformative potential of the lipreading system in real-world applications, particularly in domains such as assistive technology and human-computer interaction, where it can significantly improve communication accessibility and inclusivity for individuals with hearing impairments.

A Mimic Fusion Algorithm for Dual Channel Video Based on Possibility Distribution Synthesis Theory

In response to the current practical fusion requirements for infrared and visible videos, which often involve collaborative fusion of difference feature information, and model cannot dynamically adjust the fusion strategy according to the difference between videos, resulting in poor fusion performance, a mimic fusion algorithm for infrared and visible videos based on the possibility distribution synthesis theory is proposed. Firstly, quantitatively describe the various difference features and their attributes of the region of interest in each frame of the dual channel video sequence, and select the main difference features corresponding to each frame. Secondly, the pearson correlation coefficient is used to measure the correlation between any two features and obtain the feature correlation matrix. Then, based on the similarity measure, the fusion effective degree distribution of each layer variables for different difference features is constructed, and the difference feature distribution is correlated and synthesized based on the possibility distribution synthesis theory. Finally, optimize the select of mimic variables to achieve mimic fusion of infrared and visible videos. The experimental results show that the proposed method achieve significant fusion results in preserving targets and details, and was significantly superior to other single fusion methods in subjective evaluation and objective analysis.

GPT-4V Takes the Wheel: Promises and Challenges for Pedestrian Behavior Prediction

Predicting pedestrian behavior is the key to ensure safety and reliability of autonomous vehicles. While deep learning methods have been promising by learning from annotated video frame sequences, they often fail to fully grasp the dynamic interactions between pedestrians and traffic, crucial for accurate predictions. These models also lack nuanced common sense reasoning. Moreover, the manual annotation of datasets for these models is expensive and challenging to adapt to new situations. The advent of Vision Language Models (VLMs) introduces promising alternatives to these issues, thanks to their advanced visual and causal reasoning skills. To our knowledge, this research is the first to conduct both quantitative and qualitative evaluations of VLMs in the context of pedestrian behavior prediction for autonomous driving. We evaluate GPT-4V(ision) on publicly available pedestrian datasets: JAAD and WiDEVIEW. Our quantitative analysis focuses on GPT-4V's ability to predict pedestrian behavior in current and future frames. The model achieves a 57% accuracy in a zero-shot manner, which, while impressive, is still behind the state-of-the-art domain-specific models (70%) in predicting pedestrian crossing actions. Qualitatively, GPT-4V shows an impressive ability to process and interpret complex traffic scenarios, differentiate between various pedestrian behaviors, and detect and analyze groups. However, it faces challenges, such as difficulty in detecting smaller pedestrians and assessing the relative motion between pedestrians and the ego vehicle.

Automated Classification of Abdominal Ultrasound lmages of the Pancreas Based on the Spectral Representation of the Border’s Contours

The purpose of the research is to develop a methodology for classifying complexly structured halftone images based on a multimodal approach using methods of morphological analysis, spectral analysis and neural network modeling.Methods. А method for classifying the contours of the boundaries of segments of a complexly structured image is described. Тhe method is based on the fact that in chronic diseases of the pancreas, there is a violation of the integrity of the contour of its border and its waviness increases due to retractions and bulges caused by an alterative inflammatory process. Тhe method includes the stages of normalization of ultrasound images and image segmentation with the selection of the contour of the object of interest. Тo classify the contour of a segment boundary, it is proposed to use Fourier analysis and neural network technologies. Тhe method is illustrated using the example of classifying the contour of the border of the pancreas on its transcutaneous acoustic image.Results. Еxperimental studies of the proposed methods and means for classifying medical risk were carried out on diagnostic tasks according to the following classes: "chronic pancreatitis" – "without pathology". For experimental studies, video sequences of ultrasound images of the pancreas provided by an endoscopist were used. Тhe purpose of the experimental studies was to analyze the classification quality indicators of image classifiers with class segments "Chronic pancreatitis" and "Without pathology". Тhe training sample of video images (frames of video sequences) included 200 examples, one hundred from each class. Тhe quality indicator "Sensitivity" of classification for two classes is 85,7%, the indicator "Specificity" is 87,1%.Тhe use of the contour analysis method in classifiers of ultrasound images of the pancreas opens up new opportunities for accessible and objective diagnosis of pancreatic diseases, expanding the capabilities of intelligent clinical decision support systems.

Proximal groups: Extension of topological groups. Application in the concise representation of Hilbert envelopes on oscillatory motion waveforms

In this paper, we introduce proximal groups that are a generalization of topological groups. A straight-forward application of proximal groups is given in the concise representation of collections of overlapping Hilbert envelope lobes attached to the peak points on oscillatory motion waveforms that occur in sequences of video frames that track object movements.

Isolated Video-Based Sign Language Recognition Using a Hybrid CNN-LSTM Framework Based on Attention Mechanism

Sign language is a complex language that uses hand gestures, body movements, and facial expressions and is majorly used by the deaf community. Sign language recognition (SLR) is a popular research domain as it provides an efficient and reliable solution to bridge the communication gap between people who are hard of hearing and those with good hearing. Recognizing isolated sign language words from video is a challenging research area in computer vision. This paper proposes a hybrid SLR framework that combines a convolutional neural network (CNN) and an attention-based long-short-term memory (LSTM) neural network. We used MobileNetV2 as a backbone model due to its lightweight structure, which reduces the complexity of the model architecture for deriving meaningful features from the video frame sequence. The spatial features are fed to LSTM optimized with an attention mechanism to select the significant gesture cues from the video frames and focus on salient features from the sequential data. The proposed method is evaluated on a benchmark WLASL dataset with 100 classes based on precision, recall, F1-score, and 5-fold cross-validation metrics. Our methodology acquired an average accuracy of 84.65%. The experiment results illustrate that our model performed effectively and computationally efficiently compared to other state-of-the-art methods.

Solution of wide and micro background bias in contrastive action representation learning

In recent years, contrastive learning has made great progress in the field of computer vision, which shows great potential in action representation learning. Current contrastive learning methods usually employ contrastive loss function to learn video motion representation, which attend to capturing the similar background appearance but ignore similar motion information. This problem is called background bias, which restricts the model from exploring motion patterns. The background bias can be divided into wide background bias and micro background bias. The wide background bias refers to the statistically significant background bias, while the micro background bias refers to the background bias directly interacting with the moving object. To tackle these problems, this paper first proposes a semi-negative pair merging foreground–background, which mainly decouples the dynamic factor with obvious motion and the static factor with stable invariants in the video frame sequence. Then the dynamic factor of the original video is fused with other static factors to obtain a random background image where the foreground is more similar than the background to solve the wide background bias. Secondly, a pixel-level motion aware representation decomposition module calculates the pixel-level intensity variations from the feature space of adjacent frames, and these variations are further accumulated to obtain the salient map that focuses on the boundaries of moving objects, so the model attends to the motion pattern rather than the background to solve the micro background bias. Furthermore, a new dual loss function is proposed based on joint wide and micro background bias to better capture both static and dynamic features. The heat map shows that the proposed method effectively can solve background bias and achieve better performance in public datasets compared with other existing methods.

Scalar fields features of video sequences energy characteristics

To improve the methods of spatio-temporal filtering, an approach based on the analysis of finitary images energy spectra can be used. The purpose of this work is to study the possibility of using energy spectra for motion identification in video sequence frames. Each frame of a video sequence is a spatially finite image, the energy spectrum of which contains information about the phase components. This information is of particular importance for determining the location of dynamic objects. The most important information is also contained in the harmonic amplitude increments of the frame energy spectra, i.e. the energy spectrum generates a scalar discrete field. The energy functions describe inter-frame changes in the amplitudes of harmonics of the energy spectrum. To apply energy spectra in motion identification algorithms, it is necessary to analyze the properties of these scalar fields. Scalar fields generated by energy spectra of frames of test video sequences are studied. Test video sequences with an object moving in an arbitrary direction are used. Scalar fields of energy functions are presented graphically. Characteristic regions of scalar fields are identified and their projections are built using isolines. The boundaries of the characteristic areas and the coordinates of extrema of energy functions are determined. From them, all coordinates of the dynamic object and its horizontal/vertical displacements can be found. The dependence of phase-energy function extrema on the position of the object before the motion and on the direction of motion is studied. When moving in an arbitrary direction, different types of extrema are observed: point and extended. At vertical or horizontal movements, the energy functions graphs have only two-dimensional pulses. Additional criteria for eliminating ambiguities are proposed. The ambiguities are due to the fact that the number and nature of extrema may coincide for different directions of motion and location of objects in different quadrants of the coordinate plane. The extrema of energy functions for object movements in all possible directions and locations in any quadrant of a video sequence frame are determined. The obtained results showed that energy functions for object motion in arbitrary direction have a large number of characteristic regions containing point and extended extrema. The configuration of extrema depends both on the direction of motion and on the quadrant of the image in which the object is located. Therefore, it is most advisable to begin to determine the spatial coordinates of an object and the magnitude of its horizontal/vertical movement by finding the maximum and minimum boundaries of energy functions, and then refine it using the coordinates of the extrema. Consideration of characteristic regions and extrema of scalar fields of energy functions can be used in the development of algorithms effective in terms of "accuracy/speed" ratio, and can increase the performance of image and video processing systems. Algorithms for determining the object position, velocity and direction of its movement by analyzing scalar fields of energy spectra components can be used as an additional or main tool for motion identification in technical vision systems.

Intelligent Technique for Moving Object Detection from Problematic Video Captured through Camera Sensor

Aim: The significant aim of the proposed work is to develop an adaptive method to compute the threshold during run-time and update it adaptively for each pixel in the testing phase. It classifies motion-oriented pixels from the scene for moving objects using background subtraction and enhances using post-processing. Background: According to the huge demand for surveillance system, society is looking towards an intelligent video surveillance system that detect and track moving objects from video captured through a surveillance camera. So, it is very crucial and highly recommended throughout the globe in numerous domains such as video-based surveillance, healthcare, transportation, and many more. Practically, this research area faces lots of challenging issues such as illumination variation, cluttered background, camouflage, etc. So, this paper has developed an adaptive background subtraction method to handle such challenging problems. Objective: To focus and study the problematic video data captured through the camera sensor. To handle challenging issues available in real-time video scenes. To develop a background subtraction method and update the background model adaptively for moving object detection. Method: The proposed method has been accomplished using the following sections: Background model construction Automatic generation of threshold Background subtraction Maintenance of background model Results: The qualitative analysis of the proposed work is experimented with publicly available datasets and compared with considered state-of-the-art methods. In this work, library sequence (thermal data) of CDNET and other color video frame sequences Foreground aperture, Waving Tree and Camouflage are considered from Microsoft’s Wallflower. The quantitative values depicted in Table- 1. This work demonstrate the better performance of the proposed method as compared to state-ofthe- art methods. It also generates better outcomes and handles the problem of a dynamic environment and illumination variation. Conclusion: Currently, the world is demanding computer vision-based security and surveillancebased applications for society. This work has provided a method for the detection of moving information using an adaptive method of background subtraction approach for moving object detection in video scenes. The performance evaluation depicts better average results as compared to considered peer methods.

APPLICATION OF MACHINE LEARNING FOR RECOGNIZING SURFACE WELDING DEFECTS IN VIDEO SEQUENCES

The paper offers a solution to the problem of detecting and recognizing surface defects in welded joints that appear during tungsten inert gas welding of metal edges. This problem belongs to the machine vision. Welding of stainless-steel edges is carried out automatically on the pipe production line. Therefore, frames of video sequences are investigated. Images of some welding defects are shown in the paper. An algorithm proposed by the authors is used to detect welding defects in the video sequence frames, the efficiency of which has been confirmed experimentally. The problem solution of welding defects recognition is based on the use of traditional machine learning methods: support vector machine and artificial neural network. To build classification models, a labeled dataset containing automatically extracted texture features from the areas of welding defects detected in the video sequences was created. An analysis was performed to identify the strength of the correlation of texture features between each other and the dependent variable in the dataset for dimensionality reduction of the feature vector. The models were trained and tested on datasets with different numbers of features. The quality of the classification models was evaluated based on the accuracy metric values. The best results were achieved by the classifier built using the support vector machine with a chi-square kernel on a training sample with two features. The build models allow automatic recognition of such welding defects as lack of fusion and metal oxidation. The computational experiments with real video sequences obtained with a digital camera confirmed the possibility of using the proposed solution for recognizing surface welding defects in the process of manufacturing stainless steel pipes.

Fast Subpixel Motion Estimation Based on Human Visual System

More than 80% of video coding times are consumed by motion estimation calculations, which are the most complex aspect of the process. This method eliminates temporal redundancies in a video sequence to achieve maximum compression. Numerous efforts have been made to bring calculations closer to real time, yielding fruitful results. This study proposes a fast subpixel motion estimation algorithm for video encoding with fewer search points. This method employs the capabilities of human visual systems (HVSs), physical motion characteristics of real‐world objects, and special image information from successive frames. The number of search points (NSP) using the statistical data of the movement of the blocks in the frames of video sequences is reduced to apply fewer calculations to the system while maintaining the quality of images. Therefore, it is possible to approach fast and real‐time calculations instead of time‐consuming algorithms by accurately modeling this algorithm.

Motion-aware Dynamic Graph Neural Network for Video Compressive Sensing.

Video snapshot compressive imaging (SCI) utilizes a 2D detector to capture sequential video frames and compress them into a single measurement. Various reconstruction methods have been developed to recover the high-speed video frames from the snapshot measurement. However, most existing reconstruction methods are incapable of efficiently capturing long-range spatial and temporal dependencies, which are critical for video processing. In this paper, we propose a flexible and robust approach based on the graph neural network (GNN) to efficiently model non-local interactions between pixels in space and time regardless of the distance. Specifically, we develop a motion-aware dynamic GNN for better video representation, i.e., represent each node as the aggregation of relative neighbors under the guidance of frame-by-frame motions, which consists of motion-aware dynamic sampling, cross-scale node sampling, global knowledge integration, and graph aggregation. Extensive results on both simulation and real data demonstrate both the effectiveness and efficiency of the proposed approach, and the visualization illustrates the intrinsic dynamic sampling operations of our proposed model for boosting the video SCI reconstruction results. The code and model will be released.

Multi-View Time-Series Hypergraph Neural Network for Action Recognition.

Recently, action recognition has attracted considerable attention in the field of computer vision. In dynamic circumstances and complicated backgrounds, there are some problems, such as object occlusion, insufficient light, and weak correlation of human body joints, resulting in skeleton-based human action recognition accuracy being very low. To address this issue, we propose a Multi-View Time-Series Hypergraph Neural Network (MV-TSHGNN) method. The framework is composed of two main parts: the construction of a multi-view time-series hypergraph structure and the learning process of multi-view time-series hypergraph convolutions. Specifically, given the multi-view video sequence frames, we first extract the joint features of actions from different views. Then, limb components and adjacent joints spatial hypergraphs based on the joints of different views at the same time are constructed respectively, temporal hypergraphs are constructed joints of the same view at continuous times, which are established high-order semantic relationships and cooperatively generate complementary action features. After that, we design a multi-view time-series hypergraph neural network to efficiently learn the features of spatial and temporal hypergraphs, and effectively improve the accuracy of skeleton-based action recognition. To evaluate the effectiveness and efficiency of MV-TSHGNN, we conduct experiments on NTU RGB+D, NTU RGB+D 120 and imitating traffic police gestures datasets. The experimental results indicate that our proposed method model achieves the new state-of-the-art performance.

Video Super-Resolution Based on Inter-Frame Information Utilization for Intelligent Transportation

Intelligent transportation infrastructure is essential to intelligent transportation system (ITS). With the continuous development of Internet of Things (IoT) technology, remote monitoring has become a critical part of ITS. However, due to the limitations of network transmission, production cost, and other factors, some video monitoring can obtain only low-resolution (LR) video. LR video features are seriously lost, thus affecting the performance of ITS. In this paper, based on the research of existing super-resolution algorithms, we focus on improving the reconstruction quality of video frame sequences by aiming at the insufficient utilization of inter-frame information and low reconstruction quality of existing video super-resolution algorithms. This paper proposes a video super-resolution algorithm based on inter-frame information utilization, which can effectively improve the performance of ITS. First, a novel U-shaped feature extractor is designed to fully extract the feature expression of video frame sequences. Second, a deformable inter-frame alignment module based on residual learning is constructed to make the inter-frame alignment more accurate and thus promote the mutual utilization of inter-frame information. Finally, an up and down sampling residual block is proposed to extract features that better match the upsampling reconstruction requirements. The experimental results show that the method has better reconstruction quality for monitoring video and is advanced and applicable compared to mainstream video oversampling methods.

Enhancing temple surveillance through human activity recognition: A novel dataset and YOLOv4-ConvLSTM approach

Automated identification of human activities remains a complex endeavor, particularly in unique settings like temple environments. This study focuses on employing machine learning and deep learning techniques to analyze human activities for intelligent temple surveillance. However, due to the scarcity of standardized datasets tailored for temple surveillance, there is a need for specialized data. In response, this research introduces a pioneering dataset featuring Eight distinct classes of human activities, predominantly centered on hand gestures and body postures. To identify the most effective solution for Human Activity Recognition (HAR), a comprehensive ablation study is conducted, involving a variety of conventional machine learning and deep learning models. By integrating YOLOv4’s robust object detection capabilities with ConvLSTM’s ability to model both spatial and temporal dependencies in spatio-temporal data, the approach becomes capable of recognizing and understanding human activities in sequences of images or video frames. Notably, the proposed YOLOv4-ConvLSTM approach emerges as the optimal choice, showcasing a remarkable accuracy of 93.68%. This outcome underscores the suitability of the outlined methodology for diverse HAR applications in temple environments.

Full-resolution and full-dynamic-range coded aperture compressive temporal imaging.

Coded aperture compressive temporal imaging (CACTI) aims to capture a sequence of video frames in a single shot, using an off-the-shelf 2D sensor. This approach effectively increases the frame rate of the sensor while reducing data throughput requirements. However, previous CACTI systems have encountered challenges such as limited spatial resolution and a narrow dynamic range, primarily resulting from suboptimal optical modulation and sampling schemes. In this Letter, we present a highly efficient CACTI system that addresses these challenges by employing precise one-to-one pixel mapping between the sensor and modulator, while using structural gray scale masks instead of binary masks. Moreover, we develop a hybrid convolutional-Transformer deep network for accurate reconstruction of the captured frames. Both simulated and real data experiments demonstrate the superiority of our proposed system over previous approaches, exhibiting significant improvements in terms of spatial resolution and dynamic range.