Large Amount Of Video Data Research Articles

Convolutional neural networks for image analysis of high-speed videos from two slab burners

High-speed video recordings of slab burner experiments were analyzed using a machine learning approach with convolutional neural networks in order to compute the regression rate of hybrid rocket fuels over time. Combustion tests of paraffin-based fuel grains performed in two different hybrid rocket slab burners were recorded with high-speed video cameras and the resulting image data are analyzed in order to determine the height of the fuel in each frame. To this end, a deep neural network with U-net architecture is trained in a supervised fashion to segment the shape of the fuel slab. It is demonstrated that this approach is more capable to segment combustion images in unsteady flow conditions than classical computer vision methods based on thresholding or edge detection. Furthermore, methods in the area of uncertainty quantification of neural networks are applied to estimate the errors in the neural network prediction to new previously unseen data. Finally, the regression rate of the fuel is computed as the rate of change of this height. This method enables automatic analysis of a large amount of video data, taking full advantage of the optical access capabilities of slab burners. Additionally, the method delivers not only the time and space average values of the fuel regression rate, but also quantifies its variation over time and over the length of the slab, providing deeper insights into the combustion mechanics of hybrid rockets.

Tracking illegal activities using video surveillance systems: a review of the current state of research

The current state of research on the use of the neural networks under martial law to identify offenders committing illegal acts, prevent acts of terrorism, combat sabotage groups in cities, track weapons and control traffic is considered. The methods of detecting illegal actions, weapons, face recognition and traffic violations using video surveillance cameras are analysed. It is proposed to introduce the studied methods into the work of “smart” video surveillance systems in Ukrainian settlements.
 The most effective means of reducing the number of offences is the inevitability of legal liability for offences, so many efforts in law enforcement are aimed at preventing offences. Along with public order policing by patrol police, video surveillance is an effective way to prevent illegal activities in society. Increasing the coverage area of cameras and their number helps to ensure public safety in the area where they are used. However, an increase in the number of cameras creates another problem which is the large amount of video data that needs to be processed. To solve the problem of video data processing, various methods are used, the most modern of which is the use of artificial intelligence to filter a large amount of data from video cameras and the application of various video processing algorithms. The ability to simultaneously process video data from many CCTV cameras without human intervention not only contributes to public safety, but also improves the work of patrol police. The introduction of smart video surveillance systems allows monitoring the situation in public places around the clock, even if there is no police presence in the area.
 In the reviewed studies of video surveillance systems, neural networks, in particular MobileNet V2, YOLO, mYOLOv4-tiny, are used to track illegal actions, criminals and weapons, which are trained on large amounts of video and photo data. It has been found that although neural networks used to require a lot of computing power, they can now be used in IoT systems and smartphones, and this contributes to the fact that more video surveillance devices can be used to monitor the situation.

ConditionVideo: Training-Free Condition-Guided Video Generation

Recent works have successfully extended large-scale text-to-image models to the video domain, producing promising results but at a high computational cost and requiring a large amount of video data. In this work, we introduce ConditionVideo, a training-free approach to text-to-video generation based on the provided condition, video, and input text, by leveraging the power of off-the-shelf text-to-image generation methods (e.g., Stable Diffusion). ConditionVideo generates realistic dynamic videos from random noise or given scene videos. Our method explicitly disentangles the motion representation into condition-guided and scenery motion components. To this end, the ConditionVideo model is designed with a UNet branch and a control branch. To improve temporal coherence, we introduce sparse bi-directional spatial-temporal attention (sBiST-Attn). The 3D control network extends the conventional 2D controlnet model, aiming to strengthen conditional generation accuracy by additionally leveraging the bi-directional frames in the temporal domain. Our method exhibits superior performance in terms of frame consistency, clip score, and conditional accuracy, outperforming other compared methods.

Improving Natural Language Person Description Search from Videos with Language Model Fine-Tuning and Approximate Nearest Neighbor

Due to the ubiquitous nature of CCTV cameras that record continuously, there is a large amount of video data that are unstructured. Often, when these recordings have to be reviewed, it is to look for a specific person that fits a certain description. Currently, this is achieved by manual inspection of the videos, which is both time-consuming and labor-intensive. While person description search is not a new topic, in this work, we made two contributions. First, we improve upon the existing state-of-the-art by proposing unsupervised finetuning on the language model that forms a main part of the text branch of person description search models. This led to higher recall values on the standard dataset. The second contribution is that we engineered a complete pipeline from video files to fast searchable objects. Due to the use of an approximate nearest neighbor search and some model optimizations, a person description search can be performed such that the result is available immediately when deployed on a standard PC with no GPU, allowing an interactive search. We demonstrated the effectiveness of the system on new data and showed that most people in the videos can be successfully discovered by the search.

Deep-Learned Perceptual Quality Control for Intelligent Video Communication

With the development of video technology, a large amount of video data generated from video conferences, sports events, live broadcasts and network classes flows into our daily lives. However, ultra-high-definition video transmission is still a challenge due to the limited network bandwidth and instability, which further affects the quality of video service closely linked with consumer electronic video display. To address this challenge, we propose a deep-learned perceptual quality control approach, which can significantly improve the video quality and visual experience at the same bandwidth. The proposed scheme mainly involves saliency region extraction, perceptual-based bits allocation, and video enhancement. Firstly, we exploit a multi-scale deep convolutional network module to predict the static saliency map that semantically highlights the salient regions. Secondly, we develop a recurrent neural network model to extract the dynamic saliency regions. Finally, a three-level rate allocation scheme is developed based on the resulted saliency guidance, which is more reasonable by taking into account the visual characteristics of human eyes. Experimental results on a large dataset show that our method achieves an average gain of 1.5dB on the salient regions without introducing an extra bandwidth burden, which significantly improves the visual experience and paves the way to intelligent video communication.

Video violence recognition and localization using a semi-supervised hard attention model

The significant growth of surveillance camera networks necessitates scalable AI solutions to efficiently analyze the large amount of video data produced by these networks. As a typical analysis performed on surveillance footage, video violence detection has recently received considerable attention. The majority of research has focused on improving existing methods using supervised methods, with little, if any, attention to the semi-supervised learning approaches. In this study, a reinforcement learning model is introduced that can outperform existing models through a semi-supervised approach. The main novelty of the proposed method lies in the introduction of a semi-supervised hard attention mechanism. Using hard attention, the essential regions of videos are identified and separated from the non-informative parts of the data. A model’s accuracy is improved by removing redundant data and focusing on useful visual information in a higher resolution. Implementing hard attention mechanisms using semi-supervised reinforcement learning algorithms eliminates the need for attention annotations in video violence datasets, thus making them readily applicable. The proposed model utilizes a pre-trained I3D backbone to accelerate and stabilize the training process. The proposed model achieved state-of-the-art accuracy of 90.4% and 98.7% on RWF and Hockey datasets, respectively.

Video captioning in Vietnamese using deep learning

<p><span>With the development of today's society, demand for applications using digital cameras jumps over year by year. However, analyzing large amounts of video data causes one of the most challenging issues. In addition to storing the data captured by the camera, intelligent systems are required to quickly analyze the data to correct important situations. In this paper, we use deep learning techniques to build automatic models that describe movements on video. To solve the problem, we use three deep learning models: sequence-to-sequence model based on recurrent neural network, sequence-to-sequence model with attention and transformer model. We evaluate the effectiveness of the approaches based on the results of three models. To train these models, we use microsoft research video description corpus (MSVD) dataset including 1970 videos and 85,550 captions translated into Vietnamese. In order to ensure the description of the content in Vietnamese, we also combine it with the natural language processing (NLP) model for Vietnamese.</span></p>

Multiple Riemannian Manifold-Valued Descriptors Based Image Set Classification With Multi-Kernel Metric Learning

The importance of wild video based image set recognition is monotonically increasing due to the large amount of video data being collected by various devices including surveillance cameras, drive recorders, smart phones, and internet. The content of these videos is often complex, and it raises the question of how to perform image set modeling and feature extraction for image set-based classification. In recent years, image set classification methods have advanced considerably by modeling the image set in terms of a covariance matrix, linear subspace, or Gaussian distribution. Moreover, the distinctive geometry spanned by them include Symmetric Positive Definite (SPD) manifold, Grassmannian manifold, and Gaussian embedded Riemannian manifold, respectively. As a matter of fact, most of the approaches just adopt a single geometric model to describe each given image set, which may lose information useful for classification. To tackle this problem, we propose a novel algorithm to model each image set from a multi-geometric perspective. Specifically, the covariance matrix, linear subspace, and Gaussian distribution are applied to set representation simultaneously. In order to fuse these multiple heterogeneous Riemannian manifold-valued features, the well-equipped Riemannian kernel functions are first employed to map them into high dimensional Hilbert spaces. Then, a multi-kernel metric learning framework is devised to embed the learned hybrid kernels into a common lower dimensional subspace to facilitate classification. We conduct experiments on six widely used datasets each representing a different classification task: video-based face recognition, set-based object categorization, video-based emotion recognition, dynamic scene classification, set-based cell identification, and 3D hand pose estimation, to evaluate the classification performance of the proposed algorithm. The extensive experimental results confirm its superiority over the state-of-the-art methods.

Driver Behavior Extraction from Videos in Naturalistic Driving Datasets with 3D ConvNets

Naturalistic driving data (NDD) is an important source of information to understand crash causation and human factors and to further develop crash avoidance countermeasures. Videos recorded while driving are often included in such datasets. While there is often a large amount of video data in NDD, only a small portion of them can be annotated by human coders and used for research, which underuses all video data. In this paper, we explored a computer vision method to automatically extract the information we need from videos. More specifically, we developed a 3D ConvNet algorithm to automatically extract cell-phone-related behaviors from videos. The experiments show that our method can extract chunks from videos, most of which (~79%) contain the automatically labeled cell phone behaviors. In conjunction with human review of the extracted chunks, this approach can find cell-phone-related driver behaviors much more efficiently than simply viewing video.

SDWBF Algorithm: A Novel Pedestrian Detection Algorithm in the Aerial Scene

Due to the large amount of video data from UAV aerial photography and the small target size from the aerial perspective, pedestrian detection in drone videos remains a challenge. To detect objects in UAV images quickly and accurately, a small-sized pedestrian detection algorithm based on the weighted fusion of static and dynamic bounding boxes is proposed. First, a weighted filtration algorithm for redundant frames was applied using the inter-frame pixel difference algorithm cascading vision and structural similarity, which solved the redundancy of the UAV video data, thereby reducing the delay. Second, the pre-training and detector learning datasets were scale matched to address the feature representation loss caused by the scale mismatch between datasets. Finally, the static bounding extracted by YOLOv4 and the motion bounding boxes extracted by LiteFlowNet were subject to the weighted fusion algorithm to enhance the semantic information and solve the problem of missing and multiple detections in UAV object detection. The experimental results showed that the small object recognition method proposed in this paper enabled reaching an mAP of 70.91% and an IoU of 57.53%, which were 3.51% and 2.05% higher than the mainstream target detection algorithm.

Sports Action Recognition Based on Particle Swarm Optimization Neural Networks

Video acquisition has become more convenient as science and technology have progressed, and the development of mobile Internet has resulted in a large amount of video data being generated every day. The question of how to analyze these videos automatically has become urgent. Among them, the study of sports movement recognition in video has important theoretical implications in sports research as well as practical application value. This paper proposes a PSO-NN-based sports action recognition model. Kernel principal component analysis is used to extract and analyze the characteristics of sports movements. The improved neural network is used to identify common human postures in sports, and the classification and block background estimation method is used to detect human targets. The feature extraction of targets is completed according to the edge features, and the feature extraction of targets is completed according to the edge features. Finally, the feature vectors are trained using a backpropagation neural network (BPNN), and the parameters of the BPNN are chosen using the PSO algorithm to create a classifier for sports action recognition. The results show that this model improves the accuracy of sports video recognition and is an effective method of sports action recognition when compared to the comparison model.

A Character Recognition Tool for Automatic Detection of Social Characters in Visual Media Content

Content analysis is the go-to method for understanding how social characters, such as public figures or movie characters, are portrayed in media messages. It is an indispensable method to investigate character-related media processes and effects. However, conducting large-scale content-analytic studies is a taxing and expensive endeavor that requires hours of coder training and incurs substantial costs. This problem is particularly acute for video-based media, where coders often have to exert extensive time and energy to watch and interpret dynamic content. Here we present a Character-Recognition-Tool (CRT) that enables communication scholars to quickly process large amounts of video data to identify occurrences of specific predefined characters using facial recognition and matching. This paper presents the CRT and provides evidence for its validity. The CRT can automate the coding process of on-screen characters while following recommendations that computational tools be scalable, adaptable for novice programmers, and open source to allow for replication.

РЕАЛИЗАЦИЯ СВЕРТОЧНЫХ НЕЙРОННЫХ СЕТЕЙ НА ВСТРАИВАЕМЫХ УСТРОЙСТВАХ С ОГРАНИЧЕННЫМ ВЫЧИСЛИТЕЛЬНЫМ РЕСУРСОМ

Large amounts of video data captured by sensor sensors in various spectral ranges, the significantsize of convolutional neural network architectures create problems with the implementation ofneural network algorithms on peripheral devices due to significant limitations of computing resourceson embedded computing devices. The article discusses the use of algorithms for automatic search andpattern recognition based on machine learning methods, implemented on embedded devices with acomputing resource Graphics Processing Unit. Detection convolutional neural networks «You OnlyLook Once V3» and «You Only Look Once V3-Tiny» are used as a search and pattern recognition algorithm,which are implemented on embedded computing devices of the NVIDIA Jetson line, located indifferent price ranges and with different computing resources ... Also, in the work, the estimates ofalgorithms on embedded devices are experimentally calculated for such indicators as power consumption,forward passage time of a convolutional neural network, and detection accuracy.On the basis of solutions implemented, both at the hardware level and in software, presented byNVIDIA, it becomes possible to use deep neural network algorithms based on the convolutionoperation in real time. Computational optimization methods offered by NVIDIA are considered.Experimental studies of the influence of computations with reduced accuracy on the speed andaccuracy of object detection in images of the investigated architectures of convolutional neuralnetworks, which were previously trained on a sample of images consisting of the PASCAL VOC2007 and PASCAL VOC 2012 datasets, have been carried out.

Teleoperated Service Robot with an Immersive Mixed Reality Interface

Teleoperated service robots can perform more complex and precise tasks as they combine robot skills and human expertise. Communication between the operator and the robot is essential for remote operation and strongly affects system efficiency. Immersive interfaces are being used to enhance teleoperation experience. However, latency or time delay can impair the performance of the robot operation. Since remote visualization involves transmitting a large amount of video data, the challenge is to decrease communication instability. Then, an efficient teleoperation system must have a suitable operation interface capable of visualizing the remote environment, controlling the robot, and having a fast response time. This work presents the development of a service robot teleoperation system with an immersive mixed reality operation interface where the operator can visualize the real remote environment or a virtual 3D environment representing it. The virtual environment aims to reduce the latency on communication by reducing the amount of information sent over the network and improve user experience. The robot can perform navigation and simple tasks autonomously or change to the teleoperated mode for more complex tasks. The system was developed using ROS, UNITY 3D, and sockets to be exported with ease to different platforms. The experiments suggest that having an immersive operation interface provides improved usability for the operator. The latency appears to improve when using the virtual environment. The user experience seems to benefit from the use of mixed reality techniques; this may lead to the broader use of teleoperated service robot systems.

Saving costs for video data annotation in wildlife monitoring

In wildlife monitoring, large amounts of video data are generated by recordings from camera traps. Training of deep learning methods demands for annotated video data, i.e. video data where each frame is annotated with the correct number and species designation of the observed animals. But manual annotation of video clips is extremely time-consuming and laborious. In this proof of concept we compare three different state-of-the-art approaches to the annotation of video data: Manual annotation using the VGG Image Annotator, interactive annotation using the MiVOS video annotator and automated annotation utilizing an adapted and customized Tracktor approach that propagates annotations from frame to frame through complete video clips. An experimental proof of concept on wildlife video clips captured by camera traps show extreme time savings from hours down to minutes (i.e. in order of a magnitude) thereby not only maintaining the detection scores of animals in each frame but also improving detection scores from 54.7% to 58.5% compared to the employment of perfect but costly manual annotations in training.

A Convolution Bidirectional Long Short-Term Memory Neural Network for Driver Emotion Recognition

Real-time recognition of driver emotions can greatly improve traffic safety. With the rapid development of communication technology, it becomes possible to process large amounts of video data and identify the driver's emotions in real time. To effectively recognize driver's emotions, this paper proposes a new deep learning framework called Convolution Bidirectional Long Short-term Memory Neural Network (CBLNN). This method predicts the driver's emotion based on the geometric features extracted from facial skin information and the heart rate extracted from changes in RGB components. The facial geometry features obtained by using Convolutional Neural Network (CNN) are intermediate variables for the heart rate analysis of Bidirectional Long Short Term Memory (Bi-LSTM). Subsequently, the output of Bi-LSTM is used as input to the CNN module to extract the hear rate features. CBLNN uses Multi-modal factorized bilinear pooling (MFB) to fuse the extracted information and classifies it into five common emotions: happiness, anger, sadness, fear and neutrality. Our emotion recognition method was tested, proving that it can be used to quickly and steadily recognize emotions in real time.

Exploiting Audio-Visual Consistency with Partial Supervision for Spatial Audio Generation

Human perceives rich auditory experience with distinct sound heard by ears. Videos recorded with binaural audio particular simulate how human receives ambient sound. However, a large number of videos are with monaural audio only, which would degrade the user experience due to the lack of ambient information. To address this issue, we propose an audio spatialization framework to convert a monaural video into a binaural one exploiting the relationship across audio and visual components. By preserving the left-right consistency in both audio and visual modalities, our learning strategy can be viewed as a self-supervised learning technique, and alleviates the dependency on a large amount of video data with ground truth binaural audio data during training. Experiments on benchmark datasets confirm the effectiveness of our proposed framework in both semi-supervised and fully supervised scenarios, with ablation studies and visualization further support the use of our model for audio spatialization.

Deep learning‐based pose estimation for African ungulates in zoos

The description and analysis of animal behavior over long periods of time is one of the most important challenges in ecology. However, most of these studies are limited due to the time and cost required by human observers. The collection of data via video recordings allows observation periods to be extended. However, their evaluation by human observers is very time‐consuming. Progress in automated evaluation, using suitable deep learning methods, seems to be a forward‐looking approach to analyze even large amounts of video data in an adequate time frame.In this study, we present a multistep convolutional neural network system for detecting three typical stances of African ungulates in zoo enclosures which works with high accuracy. An important aspect of our approach is the introduction of model averaging and postprocessing rules to make the system robust to outliers.Our trained system achieves an in‐domain classification accuracy of >0.92, which is improved to >0.96 by a postprocessing step. In addition, the whole system performs even well in an out‐of‐domain classification task with two unknown types, achieving an average accuracy of 0.93. We provide our system at https://github.com/Klimroth/Video‐Action‐Classifier‐for‐African‐Ungulates‐in‐Zoos/tree/main/mrcnn_based so that interested users can train their own models to classify images and conduct behavioral studies of wildlife.The use of a multistep convolutional neural network for fast and accurate classification of wildlife behavior facilitates the evaluation of large amounts of image data in ecological studies and reduces the effort of manual analysis of images to a high degree. Our system also shows that postprocessing rules are a suitable way to make species‐specific adjustments and substantially increase the accuracy of the description of single behavioral phases (number, duration). The results in the out‐of‐domain classification strongly suggest that our system is robust and achieves a high degree of accuracy even for new species, so that other settings (e.g., field studies) can be considered.

RETRACTED ARTICLE: Simulation of English feature recognition based on machine learning and artificial intelligence technology

Computer vision includes several subjects, such as machine learning, image processing and model recognition, and replaces some human work by analyzing and extracting a large amount of video data and analyzing video content information. With the continuous improvement and development of educational video system and artificial intelligence technology, an intelligent educational video system based on real-time recognition technology is being developed rapidly. To determine and analyze the behavior of teachers and students by using behavior recognition algorithms based on continuous video images, classroom scene video can reflect students' participation in classroom teaching, and evaluate teachers' attractiveness to students through students' academic performance. Including their academic performance. Through the analysis of the behavior of teachers and students in the classroom, we can effectively find the intersection of students and teachers in the classroom. Through the interaction and interest between students, we can analyze the influence of teachers' style and interaction on students in teaching. Students, educational reform, improving the quality of education and training highly skilled personnel to achieve a positive direction.

Crowd density detection method based on crowd gathering mode and multi-column convolutional neural network

Crowds and stampedes often occur in crowd gathering places, resulting in a large number of casualties and causing great negative social impacts. Traditional research on the dynamic assessment of crowd gathering safety mainly relies on real-time video monitoring, but lacks reliable methods for processing a large amount of video data from different sources, different perspectives and different granularities. Based on Edward Hall's personal space theory, this article considers crowd psychology and other factors, and establishes static basic model of crowd gathering patterns. In order to fuse real-time multi-granularity surveillance videos with different perspectives, a multi-column convolutional neural network (M-CNN) was used to extract the local density characteristics of the crowd in a low-altitude perspective, thereby establishing a holographic model of the temporal and spatial evolution of the crowd situation, and a new crowd gathering safety assessment method. This method was actually applied to the safety assessment of crowd gathering in Suzhou landmark-Urban Living Fountain Square, and achieved good results, providing theoretical support for the safety management of crowd gathering places.