Continuous Video Frames Research Articles

Simulation of artificial intelligence robots in dance action recognition and interaction process based on machine vision

Artificial intelligence robots based on machine vision have shown great potential in dance action recognition and interaction. In order to perform action recognition, a large amount of dance action video data was collected as the basis for training and testing the model. In motion recognition, various computer vision techniques are used to extract motion features from images or videos to capture key information of dance movements. Machine learning technology was adopted to construct an action classification model. By using the data collected in the early stage and the extracted features, an efficient and accurate classifier has been studied and trained. It can classify the input dance actions into different categories, segment continuous video frames into different action segments, and ensure accurate differentiation of different dance actions during the recognition process, avoiding misjudgment or confusion. By accurately capturing and analyzing the posture information of the human body, we can better understand and simulate dance movements, and enhance the performance and realism of artificial intelligence robots in dance interactions. The experimental results show that the system can accurately recognize various dance movements and achieve good interaction with users.

YH-Pose: Human pose estimation in complex coal mine scenarios

Human pose estimation in continuous video frames captured from complex coal mine scenes is challenging. The video frames in this scene may suffer from problems such as uneven brightness, blurred image details, and excessive noise. Mainstream pose estimation methods have good detection performance for high-quality static images, but their accuracy and prediction rate can be significantly reduced in coal mine scenes. In this work, a yielding human pose estimation framework is proposed, termed YH-Pose. The framework strives to incorporate additional visual evidences from neighboring frames to facilitate the pose estimation of the current frame. Firstly, a human detector is introduced to locate the person’s position in the video frame and extract global features. This can provide a good initialization for the latter keypoint detection, making the training process converge quickly. Secondly, heatmaps are used to encode the joint locations as Gaussian peaks, and a temporal road module (TRM) is designed, which encodes video frames at intervals. The module efficiently fuses spatio-temporal information through frame-rate groups in a hierarchical manner. Lastly, the spatial road module (SRM) learns from fused keypoint context features and predicts the location of the keypoints in the next frame. In addition, a dataset called Colliery-1 is proposed, which derives from underground surveillance video from chinese coal mines and consists of 3600 video clips. The experimental results on the Colliery-1 dataset indicate that the framework achieved an average accuracy of 82% and 80% on the training and test sets, respectively. Moreover, the framework achieved a 94.2% prediction rate for pose estimation. To further evaluate the effectiveness of the proposed method, some comparisons have been made between it and various mainstream methods using different metrics.

Tracking by Joint Local and Global Search: A Target-Aware Attention-Based Approach.

Tracking-by-detection is a very popular framework for single-object tracking that attempts to search the target object within a local search window for each frame. Although such a local search mechanism works well on simple videos, however, it makes the trackers sensitive to extremely challenging scenarios, such as heavy occlusion and fast motion. In this article, we propose a novel and general target-aware attention mechanism (termed TANet) and integrate it with a tracking-by-detection framework to conduct joint local and global search for robust tracking. Specifically, we extract the features of the target object patch and continuous video frames; then, we concatenate and feed them into a decoder network to generate target-aware global attention maps. More importantly, we resort to adversarial training for better attention prediction. The appearance and motion discriminator networks are designed to ensure its consistency in spatial and temporal views. In the tracking procedure, we integrate target-aware attention with multiple trackers by exploring candidate search regions for robust tracking. Extensive experiments on both short- and long-term tracking benchmark datasets all validated the effectiveness of our algorithm.

Abnormal Behavior Learning Based on Edge Computing toward a Crowd Monitoring System

Abnormal behavior poses a great threat to social security and stability. The resulting violence or crime leads to terrible consequences. How to utilize reasonable means to predict the dangerous intentions of massive crowds and prevent the potential hazard to the public is significant for social security. A crowd monitoring and management system is an effective way to detect abnormal behavior. In this article, we release unmanned aerial vehicles as well as fixed ground devices to achieve multi-level and multi-modal behavioral sensing on a massive crowd, deploy a hybrid model in edge cloud to extract global features from behavioral data of a massive crowd, and then utilize these global features to construct decent classification algorithms for action recognition and behavioral semantic cognition. With the cooperation of behavioral data and cognitive algorithms, we can understand the instantaneous emotions of the crowd. On the basis of behavioral data and the emotional state of the crowd, the correlation between daily behavior and any dangerous intention of a massive crowd has been revealed by utilizing behavioral big data analysis, which is a key foundation for predicting people's dangerous intentions. Finally, we conduct a case study of abnormal behavior detection based on pix2pix and continuous video frames. The experimental results show that the performance of our method is better than other algorithms in both public datasets and the customized Hajj dataset. The proposed novel pattern for the effective learning of a massive crowd is validated to effectively eliminate some of the possible dangers caused by abnormal behavior.

Self-Teaching Video Object Segmentation.

Video object segmentation (VOS) is one of the most fundamental tasks for numerous sequent video applications. The crucial issue of online VOS is the drifting of segmenter when incrementally updated on continuous video frames under unconfident supervision constraints. In this work, we propose a self-teaching VOS (ST-VOS) method to make segmenter to learn online adaptation confidently as much as possible. In the segmenter learning at each time slice, the segment hypothesis and segmenter update are enclosed into a self-looping optimization circle such that they can be mutually improved for each other. To reduce error accumulation of the self-looping process, we specifically introduce a metalearning strategy to learn how to do this optimization within only a few iteration steps. To this end, the learning rates of segmenter are adaptively derived through metaoptimization in the channel space of convolutional kernels. Furthermore, to better launch the self-looping process, we calculate an initial mask map through part detectors and motion flow to well-establish a foundation for subsequent refinement, which could result in the robustness of the segmenter update. Extensive experiments demonstrate that this ST idea can boost the performance of baselines, and in the meantime, our ST-VOS achieves encouraging performance on the DAVIS16, Youtube-objects, DAVIS17, and SegTrackV2 data sets, where, in particular, the accuracy of 75.7% in J-mean metric is obtained on the multi-instance DAVIS17 data set.

Sound Can Help Us See More Clearly.

In the field of video action classification, existing network frameworks often only use video frames as input. When the object involved in the action does not appear in a prominent position in the video frame, the network cannot accurately classify it. We introduce a new neural network structure that uses sound to assist in processing such tasks. The original sound wave is converted into sound texture as the input of the network. Furthermore, in order to use the rich modal information (images and sound) in the video, we designed and used a two-stream frame. In this work, we assume that sound data can be used to solve motion recognition tasks. To demonstrate this, we designed a neural network based on sound texture to perform video action classification tasks. Then, we fuse this network with a deep neural network that uses continuous video frames to construct a two-stream network, which is called A-IN. Finally, in the kinetics dataset, we use our proposed A-IN to compare with the image-only network. The experimental results show that the recognition accuracy of the two-stream neural network model with uesed sound data features is increased by 7.6% compared with the network using video frames. This proves that the rational use of the rich information in the video can improve the classification effect.

Channel-Wise Spatiotemporal Aggregation Technology for Face Video Forensics

Recent progress in deep learning, in particular the generative models, makes it easier to synthesize sophisticated forged faces in videos, leading to severe threats on social media about personal privacy and reputation. It is therefore highly necessary to develop forensics approaches to distinguish those forged videos from the authentic. Existing works are absorbed in exploring frame-level cues but insufficient in leveraging affluent temporal information. Although some approaches identify forgeries from the perspective of motion inconsistency, there is so far not a promising spatiotemporal feature fusion strategy. Towards this end, we propose the Channel-Wise Spatiotemporal Aggregation (CWSA) module to fuse deep features of continuous video frames without any recurrent units. Our approach starts by cropping the face region with some background remained, which transforms the learning objective from manipulations to the difference between pristine and manipulated pixels. A deep convolutional neural network (CNN) with skip connections that are conducive to the preservation of detection-helpful low-level features is then utilized to extract frame-level features. The CWSA module finally makes the real or fake decision by aggregating deep features of the frame sequence. Evaluation against a list of large facial video manipulation benchmarks has illustrated its effectiveness. On all three datasets, FaceForensics++, Celeb-DF, and DeepFake Detection Challenge Preview, the proposed approach outperforms the state-of-the-art methods with significant advantages.

AI 딥러닝을 이용한 싸움(다툼) 행동 인식 방법

In this study, we developed an AI deep learning-based fighting behavior recognition method for a video surveillance system and proved its effectiveness through various experiments. The proposed method consists of a two-step fighting behavior recognition framework. First, continuous video frames of the target surveillance video are transmitted to the Inflated 3D ConvNet (I3D) network, which shows a good behavior-recognition performance, to extract the spatiotemporal features. These extracted 3D features are then used as the inputs in the next step, where a fight situation is detected using a classification model consisting of a fully connected layer. To use the proposed aggressive behavior detection framework effectively, first, it is necessary to train the fight detection model. However, it is not possible to collect sufficient fighting videos in various outdoor environments. To overcome this limitation, we generated a large amount of learning data through data augmentation. Therefore, instead of directly learning from the training videos transmitted to the I3D network, the classifier trains itself to recognize various fighting actions using the Kinetics video dataset. That is, the action features are extracted from the transmitted consecutive frames using the pretrained I3D network and subsequently used to train the fully connected layer classification model. In addition, we proposed a learning method that includes recognizing ambiguous conflict boundaries using multiple instance learning to mitigate the ambiguous starting and ending of the contention videos. The effectiveness of the proposed method was verified through several experiments by drawing comparisons between the present results and those of the previously reported studies.

Video Decolorization Based on the CNN and LSTM Neural Network

Video decolorization is the process of transferring three-channel color videos into single-channel grayscale videos, which is essentially the decolorization operation of video frames. Most existing video decolorization algorithms directly apply image decolorization methods to decolorize video frames. However, if we only take the single-frame decolorization result into account, it will inevitably cause temporal inconsistency and flicker phenomenon meaning that the same local content between continuous video frames may display different gray values. In addition, there are often similar local content features between video frames, which indicates redundant information. To solve the preceding problems, this article proposes a novel video decolorization algorithm based on the convolutional neural network and the long short-term memory neural network. First, we design a local semantic content encoder to learn and extract the same local content of continuous video frames, which can better preserve the contrast of video frames. Second, a temporal feature controller based on the bi-directional recurrent neural networks with Long short-term memory units is employed to refine the local semantic features, which can greatly maintain temporal consistency of the video sequence to eliminate the flicker phenomenon. Finally, we take advantages of deconvolution to decode the features to produce the grayscale video sequence. Experiments have indicated that our method can better preserve the local contrast of video frames and the temporal consistency over the state of the-art.

Attention-guided dual spatial-temporal non-local network for video super-resolution

In this paper, we propose an attention-guided dual spatial-temporal non-local network for video super-resolution (ADNLVSR). We integrate temporal and spatial non-local self-similar contexts from continuous video frames after motion compensation, and merge the features of different levels discriminatively with channel attention mechanism for target frame. During motion compensation, unlike previous methods directly stacking input images or features for merging, we use learnable attention mechanism to guide the merging, which suppresses undesired components caused by misalignment and enhances desirable fine details. During feature fusion, in contrast to most previous approaches where global-level non-local self-similarity existing in space or time is usually considered, we propose region-level spatial and temporal non-local operations for exploiting temporal correlations and enhancing similar spatial structures. The proposed modules can effectively avoid the computational burden caused by existing global-level non-local operations based on our analysis, and enhance correlated structure information. In addition, we propose a channel attention-guided residual dense block (CRDB), in which a second-order channel attention mechanism is applied to adaptively rescale the channel-wise features for more discriminative representations. Extensive experiments on different datasets demonstrate superior performance to state-of-the-art published methods on video super-resolution.

A Shallow Prediction Model with Enhanced Features for Video Time Series

Fast video time series prediction is an important part in video anomaly detection in deep learning framework. Time series deep learning models have good performance in time-series data prediction, such as traffic flow, rainfall and network traffic. The models, such as ConvLSTM, ConvGRU with complex structure and large amount of computation, are difficult to fit the video time series prediction. In this paper, we analyze data characteristics of video sequences and propose a shallow video series prediction model. We not only consider the spatial correlation in continuous video frames, but also investigate the interval video frames, which also maintain spatial correlation. We explore the gray scale pixel values of the interval frames which appear specific trend properties when we connect the context pixels together. Based on the enhanced features, we design a shallow 2D convolutional video time series prediction model with good prediction performance and much less total parameters.

Towards Highly Accurate and Stable Face Alignment for High-Resolution Videos

In recent years, heatmap regression based models have shown their effectiveness in face alignment and pose estimation. However, Conventional Heatmap Regression (CHR) is not accurate nor stable when dealing with high-resolution facial videos, since it finds the maximum activated location in heatmaps which are generated from rounding coordinates, and thus leads to quantization errors when scaling back to the original high-resolution space. In this paper, we propose a Fractional Heatmap Regression (FHR) for high-resolution video-based face alignment. The proposed FHR can accurately estimate the fractional part according to the 2D Gaussian function by sampling three points in heatmaps. To further stabilize the landmarks among continuous video frames while maintaining the precise at the same time, we propose a novel stabilization loss that contains two terms to address time delay and non-smooth issues, respectively. Experiments on 300W, 300VW and Talking Face datasets clearly demonstrate that the proposed method is more accurate and stable than the state-ofthe-art models.

Inter-frame video image generation based on spatial continuity generative adversarial networks

This paper proposes a method for generating inter-frame video images based on spatial continuity generative adversarial networks (SC-GANs) to smooth the playing of low-frame rate videos and to clarify blurry image edges caused by the use of traditional methods to improve the video frame rate. Firstly, the auto-encoder is used as a discriminator and Wasserstein distance is applied to represent the difference between the loss distribution of the real sample and the generated sample, instead of the typical method of generative adversarial networks to directly match data distribution. Secondly, the hyperparameter between generator and discriminator is used to stabilize the training process, which effectively prevents the model from collapsing. Finally, taking advantage of the spatial continuity of the image features of continuous video frames, an optimal value between two consecutive frames is found by Adam and then mapped to the image space to generate inter-frame images. In order to illustrate the authenticity of the generated inter-frame images, PSNR and SSIM are adopted to evaluate the inter-frame images, and the results show that the generated inter-frame images have a high degree of authenticity. The feasibility and validity of the proposed method based on SC-GAN are also verified.

Higher Pollution Episode Detection Using Image Classification Techniques

Image classification techniques have been applied to detect higher pollution episodes in modelled air pollution data. These techniques are widely used in video processing to find patterns in videos. An attempt for the first time has been made to apply these techniques by considering air pollution as continuous video frames as the spatio-temporal changes in the pollution are linked to its previous state of the atmosphere. The applicability of these techniques has been tested over Northern Italy to detect ozone pollution episodes in year 2004 using model simulated concentrations. The methods tested in this paper are pixel, block-based, histogram, pertinent pixel and twin-comparison method. While these techniques have some kind of merits and demerits, a modified pertinent pixel comparison algorithm has been proposed to detect pollution episodes. The proposed method has been validated to detect PM10 episodes over Milan metropolitan area during 2 months in 2008 and is able to detect PM10 episodic events as well as non-events. This method provides a single binary index that can be applied by the air quality modellers and decision makers to determine the pollution episode over a given domain.

Design and Implementation of TrackingSystem

Locating an object of interest in continuous video frames can be termed as Object Tracking. Computer Vision is branch recently focussing on interaction between human and computer system. Continuously tracking desired object in consecutive frames is a difficult task. Tracking helps in sorting out situations such criminal incidents, traffic congestion, reckless driving and speed violation. The object in motion is detected using algorithm based on frame differencing and background subtraction. Morphological operations along with filtering are implemented to eliminate noise. The object co-ordinates are displayed along with direction of motion. It is a real time human tracking System. The same algorithm is used for color (yellow) based tracking. For vehicle tracking Simulink model is implemented in which stored video is processed using frame differencing and optical flow. Thus using this algorithm continuous monitoring and surveillance system is introduced.

Learning discriminative features for fast frame-based action recognition

In this paper we present an instant action recognition method, which is able to recognize an action in real-time from only two continuous video frames. For the sake of instantaneity, we employ two types of computationally efficient but perceptually important features – optical flow and edges – to capture motion and shape characteristics of actions. It is known that the two types of features can be unreliable or ambiguous due to noise and degradation of video quality. In order to endow them with strong discriminative power, we pursue combined features, of which the joint distributions are different in-between action classes. As the low-level visual features are usually densely distributed in video frames, to reduce computational expense and induce a compact structural representation, we propose to first group the learned discriminative joint features into feature groups according to their correlation, then adapt the efficient boosting method as the action recognition engine which take the grouped features as input. Experimental results show that the combination of the two types of features achieves superior performance in differentiating actions than that of using each single type of features alone. The whole model is computationally efficient, and the action recognition accuracy is comparable to the state-of-the-art approaches.

관심 영역의 트랜스코딩 기법을 이용한 모바일 프리젠테이션

An effective integration of web-based learning environment and mobile device technology is considered as a new challenge to the developers. The screen size, however, of the mobile device is too small, and its performance is too inferior. Due to the foregoing limit of mobile technology, displaying bulk data on the mobile screen, such as a cyber lecture accompanied with real-time image transmission on the web, raises a lot of problems. Users have difficulty in recognizing learning contents exactly by means of a mobile device, and continuous transmission of video stream with bulky information to the mobile device arouses a lot of load for the mobile system. Thus, an application which is developed to be applied in PC is improper to be used for the mobile device as it is, a player which is fitting for the mobile device should be developed. Accordingly, this paper suggests mobile presentation using transcoding techniques of the field concerned. To display continuous video frames of learning image, such as a cyber lecture or remote lecture, by means of a mobile device, the performance difference between high-resolution digital image and mobile device should be surmounted. As the transcoding techniques to settle the performance difference causes damage of image quality, high-quality image may be guaranteed by application of trial and error between transcoding and selected learning resources.