Complex Video Research Articles

Development of an advanced method for compressing the dynamic component of a differential-represented frame

For a typical low complexity video sequence, the weight of each P-frame in the stream is approximately three times smaller than the I-frame weight. However, taking into account the number of P-frames in the group, they make the main contribution to the total video data amount. Therefore, the possibility of upgrading coding methods for P-frames is considered on preliminary blocks' type identification with the subsequent formation of block code structures. As the correlation coefficient between adjacent frames increases, the compression ratio of the differential-represented frame's binary mask increases. The compression ratio of the differential-represented frame's binary mask varies from 3 to 21 depending on the correlation coefficient between adjacent frames. Estimation of the bit representation's information content of the differential-represented frame's binary mask on the basis of accounting for the nonequilibrium of the bases of the lengths of the binary series does not require an increase in the complexity of the software-hardware implementation. Due to the double-alphabetic power code, the differential-represented frame's binary mask is relative to the single-alphabet code will decrease by 17%. The most preferable method for constructing the compact representation technology of the binary masks of frames represented in a differential form is the approach. This is based on the identification and description of the lengths of one-dimensional binary series. A binary series is a consecutive binary elements sequence with the same value. In this case, sequences of identical binary elements are replaced by their lengths. And since the elements of the binary masks of the frames represented in the differential form take only two possible values 0 or 1, it is suggested to form the lengths of the binary series without indicating their level.

A cascaded CNN model for multiple human tracking and re-localization in complex video sequences with large displacement

Human tracking and localization play a crucial role in many applications like accident avoidance, action recognition, safety and security, surveillance and crowd analysis. Inspired by its use and scope, we introduced a novel method for human tracking (one or many) and re-localization in a complex environment with large displacement. The model can handle complex background, variations in illumination, changes in target pose, the presence of similar target and appearance (pose and clothes), the motion of target and camera, occlusion of the target, background variation, and massive displacement of the target. Our model uses three convolutional neural network based deep architecture and cascades their learning such that it improves the overall efficiency of the model. The first network learns the pixel level representation of small regions. The second architecture uses these features and learns the displacement of a region with its category between moved, not-moved, and occluded classes. Whereas, the third network improves the displacement result of the second network by utilizing the previous two learning. We also create a semi-synthetic dataset for training purpose. The model is trained on this dataset first and tested on a subset of CamNeT, VOT2015, LITIV-tracking and Visual Tracker Benchmark database without training with real data. The proposed model yield comparative results with respect to current state-of-the-art methods based on evaluation criteria described in Object Tracking Benchmark, TPAMI 2015, CVPR 2013 and ICCV 2017.

Location, Integration, Interruption: Visual Properties and Recognition of Video Game Advertising

Videogame placements are significant forms of advertising. There is, however, a limited amount of cognitive capacity available to process messages at any given moment during a complex video game. Using the Limited Capacity Model as an exploratory mechanism, it is proposed that the recall and recognition of in-game advertising depends on its positioning in the field-of-view and its integration within the game world. This is a quasi-experimental study involving 42 Arab women, key purchasing decision-makers and high-level consumers of luxury products, playing a commercial driving videogame. Participants were questioned about message recognition after 40 minutes of play. Placement positions in the center of the screen enhanced brand and message recognition. Ads integrated into the game environment were better recognized than virtual billboards. Landmarks were better recognized than either integrated ads or billboards.

Weakly supervised deep network for spatiotemporal localization and detection of human actions in wild conditions

Human action localization in any long, untrimmed video can be determined from where and what action takes place in a given video segment. The main hurdles in human action localization are the spatiotemporal randomnesses of their happening in a parallel mode which means the location (a particular set of frames containing action instances) and duration of any particular action in real-life video sequences generally are not fixed. At another end, the uncontrolled conditions such as occlusions, viewpoints and motions at the crisp boundary of the action sequences demand to develop a fast deep network which can be easily trained from unlabeled samples of complex video sequences. Motivated from the facts, we proposed a weakly supervised deep network model for human action localization. The model is trained from unlabeled action samples from UCF50 action benchmark. The five-channel data obtained from the concatenation of RGB (three-channel) and optical flow vectors (two-channel) are fed to the proposed convolutional neural network. LSTM network is used to yield the region of action happening area. The performance of the model is tested on UCF-sports dataset. The observation and comparative results reflect that our model can localize any action from annotation-free data samples captured in uncontrolled conditions.

Computer processing of distorted video sequences obtained by mobile road laboratories

One of the most promising road TPC diagnosis areas is computer processing of data obtained by mobile road laboratories equipped with global positioning systems (GPS, GLONASS), inertial navigation systems (INS), complex video and other devices. The aim of the research is to develop new technologies used for computer processing of video sequences obtained by mobile road laboratories equipped with video-recording systems providing minimization of energy and labor costs. The article presents the results of research on the development of software used for processing video sequences obtained by mobile road laboratories equipped with video-recording systems. Works on the use of video processing software published for the last five years were reviews.

Self-Motion-Assisted Tensor Completion Method for Background Initialization in Complex Video Sequences.

The background Initialization (BI) problem has attracted the attention of researchers in different image/video processing fields. Recently, a tensor-based technique called spatiotemporal slice-based singular value decomposition (SS-SVD) has been proposed for background initialization. SS-SVD applies the SVD on the tensor slices and estimates the background from low-rank information. Despite its efficiency in background initialization, the performance of SS-SVD requires further improvement in the case of complex sequences with challenges such as stationary foreground objects (SFOs), illumination changes, low frame-rate, and clutter. In this paper, a self-motion-assisted tensor completion method is proposed to overcome the limitations of SS-SVD in complex video sequences and enhance the visual appearance of the initialized background. With the proposed method, the motion information, extracted from the sparse portion of the tensor slices, is incorporated with the low-rank information of SS-SVD to eliminate existing artifacts in the initiated background. Efficient blending schemes between the low-rank (background) and sparse (foreground) information of the tensor slices is developed for scenarios such as SFO removal, lighting variation processing, low frame-rate processing, crowdedness estimation, and best frame selection. The performance of the proposed method on video sequences with complex scenarios is compared with the top-ranked state-of-the-art techniques in the field of background initialization. The results not only validate the improved performance over the majority of the tested challenges but also demonstrate the capability of the proposed method to initialize the background in less computational time.

Exploiting Mid-Level Semantics for Large-Scale Complex Video Classification

As the amount of available video data has grown substantially, automatic video classification has become an urgent yet challenging task. Most video classification methods focus on acquiring discriminative spacial visual features and motion patterns for video representation, especially deep learning methods, which have achieved very good results on action recognition problems. However, the performance of most of these methods drastically degenerates for more generic video classification tasks where the video contents are much more complex. Thus, in this paper, the mid-level semantics of videos are exploited to bridge the semantic gap between low-level features and high-level video semantics. Inspired by the term ``frequency-inverse document frequency'', a word weighting method for the problem of text classification is introduced to the video domain. The visual objects in videos are regarded as the words in texts, and two new weighting methods are proposed to encode videos by weighting visual objects according to the characteristics of videos. In addition, the semantic similarities between video categories and visual objects are introduced from the text domain as privileged information to facilitate classifier training on the obtained semantic representations of videos. The proposed semantic encoding method (semantic stream) is then fused with the popular two-stream CNN model for the final classification results. Experiments are conducted on two large-scale complex video datasets, CCV and ActivityNet. The experimental results validate the effectiveness of the proposed methods.

Structured Learning for Action Recognition in Videos

Actions in continuous videos are correlated and may have hierarchical relationships. Densely labeled datasets of complex videos have revealed the simultaneous occurrence of actions, but existing models fail to make use of the relationships to analyze actions in the context of videos and better understand complex videos. We propose a novel architecture consisting of a correlation learning and input synthesis (CoLIS) network, long short-term memory (LSTM), and a hierarchical classifier. First, the CoLIS network captures the correlation between features extracted from video sequences and pre-processes the input to the LSTM. Since the input becomes the weighted sum of multiple correlated features, it enhances the LSTM’s ability to learn variable-length long-term temporal dependencies. Second, we design a hierarchical classifier which utilizes the simultaneous occurrence of general actions such as run and jump to refine the prediction on their correlated actions. Third, we use interleaved backpropagation through time for training. All these networks are fully differentiable so that they can be integrated for end-to-end learning. The results show that the proposed approach improves action recognition accuracy by 1.0% and 2.2% on single-labeled or densely labeled datasets respectively.

Object removal from complex videos using a few annotations

We present a system for the removal of objects from videos. As input, the system only needs a user to draw a few strokes on the first frame, roughly delimiting the objects to be removed. To the best of our knowledge, this is the first system allowing the semi-automatic removal of objects from videos with complex backgrounds. The key steps of our system are the following: after initialization, segmentation masks are first refined and then automatically propagated through the video. Missing regions are then synthesized using video inpainting techniques. Our system can deal with multiple, possibly crossing objects, with complex motions, and with dynamic textures. This results in a computational tool that can alleviate tedious manual operations for editing high-quality videos.

Foreground Segmentation Using Adaptive 3 Phase Background Model

An extensive collection of algorithms have been proposed over the years to identify the foreground from a video scene, but none of them considers the classification history of previous frames for discovering moving objects. All the existing algorithms focus on a single background model to cope with all types of challenging and complex video environments. In this paper, a real-time pixel level classification method is proposed that uses its previous output history to update its parameters. The model has three components designed to handle various types of challenging background environments. For each pixel, three sub-parts, namely the Adaptive Background Model, the Neighborhood Background Model, and the Change Detection Background Model are constructed to detect various types of complex background changes. A pixel level model updating method uses the previous foreground/background binary classification results to periodically refresh all the three background models. This update method helps to recognize the foreground accurately by adjusting the algorithmic parameters to efficiently detect complex background changes. This novel method shows a significant improvement in performance for a variety of complex video scenes.

Hierarchical feature representation for unconstrained video analysis

Complex video analysis is a challenging problem due to the long and sophisticated temporal structure of unconstrained videos. This paper introduces pooled-feature representation (PFR) which is derived from a double layer encoding framework (DLE) to address this problem. Considering that a complex video is composed of a sequence of simple frames, the first layer generates temporal sub-volumes from the video and represents them individually. The second layer constructs the pool of features by fusing the represented vectors from the first layer. The pool is compressed and then encoded to provide video-parts vector (VPV). This framework allows distilling the representation and extracting new information in a hierarchical way. Compared with recent video encoding approaches, VPV can preserve the higher-level information through typical encoding in the higher layer. Furthermore, the encoded vectors from both layers of DLE are fused along with a compression stage to develop PFR. The early and late fusion stages are adopted based on the priority of compression stage over concatenation of represented vectors. To validate the proposed framework, we conduct extensive experiments on four complex action datasets: UCF50, HMDB51, URADL, and Olympic. Experimental results demonstrate that PFR with early fusion achieves the state-of-the-art performance by capturing the most prominent features with minimum dimension.

Spatiotemporal local compact binary pattern for background subtraction in complex scenes

A variety of binary feature descriptors such as local binary pattern (LBP) and its variations have recently attracted considerable attention for modelling backgrounds, due to their robustness and strong discriminatory power. However, most existing binary feature descriptors fail to model complex scenes due to their sensitivity to noise. In this paper, we propose an effective local compact binary descriptor for background modelling. For each image, local compact binary patterns (LCBPs) are first extracted by computing a number of low-dimensional pixel difference vectors (PDVs). Then, the LCBP is extended to the spatiotemporal domain taking into account the temporal persistence of pixels, and a novel local compact binary descriptor, STLCBP, is proposed. Multiple color spaces are also considered in order to separate foreground from background pixels accurately. Finally, a joint domain-range adaptive kernel density estimate (KDE) model is used to estimate the background and foreground scores by combining texture features with color features. Experimental results on two well-known datasets, I2R and CDnet2014, demonstrate that the proposed approach significantly outperforms many state-of-the-art methods and works effectively on a wide range of complex videos.

Crowd behavior analysis using MoDTA approach

<span>In order to analyze the behaviors of human, significant extent of work has been carried out in the video surveillance applications. While considering the crowded scenes, the adopted features are crafted manually which have a great side to detect anomaly. It requires prior information and is hard to extract from complex video scenes and also it involves huge computational costs. In this paper, we are proposing multi-observational detection and tracking approach (MoDTA) that is based on observational filter. The MoDTA initially acquires<span> </span>people location in an image, </span>so that is <span>can detect conviction value at pointed locations which generally increases with respect to people density. In the phase of tracking, MoDTA computes the multiple observed weight values and individual features, also advection particle is used at motion model in order to facilitate the dense scenario tracking. Coefficient of correlation is used as template detector and the function of template detector is to estimate the upcoming object. Our proposed MoDTA is compared with other existing detection and tracking methods in order to evaluate the system performance.<span> </span></span>

Action and movements recognition methods

The article describes the formulation of the problem of recognition of the movements of objects in a video sequence, the stages of its solution, the analysis of the basic methods of each of the stages. A wide range of applications and growing requirements on the quality of recognition determines the relevance of the study. The process of action recognition and detection begins with extracting useful features, from the input video sequence. Features are then processed through a classier to identify the action class (for example, running, walking, jumping, various gestures). The article describes the main feature descriptors, in the filter-based category: histogram of oriented gradients, cuboid descriptor, scale-invariant feature transform, gradient location-orientation histogram, local trinary patterns, and spatiotemporal patches, optical flow-based descriptors: histograms of optical flow, the motion boundary histogram, dense trajectory, convolutional neural network-based descriptors. Some algorithms require the extraction of primitive features and further refinement of the auxiliary features before they can be passed to the classifier. Examples of the use of specialized primitive features are methods based on silhouettes / contours and methods based on object tracking. There are methods for classifying extracted features, including the following: support vector machines, adaptive boost, artificial neural networks, convolutional neural networks. The key difficulties arising in solving the problem are considered. There are ways to compare various methods. One of the ways to draw comparisons is to quantitatively evaluate each approach on the same database with the same protocol. From simple KTH datasets and Weizmannnd to Carnegie Mellon University Crowded Videos dataset and Microsoft Research Action Group dataset to more complex video conditions and large-scale UCF101 and ActivityNet datasets. Existing approaches to recognition of motion in video sequences are analyzed. The article reveals characteristics, strengths and weaknesses of the various methods of detecting features and their classification. Leading methods that show the best results widely use convolutional neural networks. One of such methods is a spatio-temporal graph convolutional neural network for action recognition based on the object's skeleton. A method for further research and improvement was chosen.

STRNN: End-to-end deep learning framework for video partial copy detection

The task of partial copy detection in videos aims at determine if one or more segments of the query video are already present in the data-set, while giving the information of similar portion time period. At present, most effective algorithms of partial copy detection in videos are designed as three steps: feature extraction, feature matching and time alignment. The separation of feature matching and time alignment module ignores the spatio-temporal information of partial copy to some extent. Therefore, satisfactory performance is not obtained. In order to reduce this loss, this article does not decompose it into two separate tasks, but using a single convolution neural network to solve these two aspects. First, we sample video frames and extract CNN features, calculate the spatio-temporal relationship matrix of the source video and the query video, and then graphically map the matrix and train the convolution neural network based on the object detection task of the RefineDet model. Finally, in the query phase, the time period of the partial copy is deduced based on the detection result. In this paper, we evaluate the performance of the algorithm on the real complex video copy detection data-set VCDB which is significantly improved compared with the state-of-the-art partial copy detection framework.

An Overview of Emerging Video Coding Standards

Today's popular video coding standards, such as H.264/AVC, are widely used to encode video into bit streams for storage and transmission. With the explosive growth of various video applications, H.264/AVC may not fully satisfy their requirements anymore. There is an increasing demand for high compression efficiency and low complexity video coding standards. In this article, we provide an overview of existing and emerging video coding standards. We review the timeline of the development of the popular H.26X family video coding standards, and introduce several emerging video coding standards such as AV1, VP9 and VVC. As for future video coding, considering the success of machine learning in various fields and hardware acceleration, we conclude this article with a discussion of several future trends in video coding.

Engineering a Live UHD Program from the International Space Station

The first-ever live downlink of Ultra-High Definition (UHD) video from the International Space Station (ISS) was the highlight of a “Super Session” at the National Association of Broadcasters (NAB) Show in April 2017. Ultra-High Definition is four times the resolution of “full HD” or “1080P” video. Also referred to as “4K”, the Ultra-High Definition video downlink from the ISS all the way to the Las Vegas Convention Center required considerable planning, pushed the limits of conventional video distribution from a space-craft, and was the first use of High Efficiency Video Coding (HEVC) from a space-craft. The live event at NAB will serve as a pathfinder for more routine downlinks of UHD as well as use of HEVC for conventional HD downlinks to save bandwidth. A similar demonstration was conducted in 2006 with the Discovery Channel to demonstrate the ability to stream HDTV from the ISS. This paper will describe the overall work flow and routing of the UHD video, how audio was synchronized even though the video and audio were received many seconds apart from each other, and how the demonstration paves the way for not only more efficient video distribution from the ISS, but also serves as a pathfinder for more complex video distribution from deep space. The paper will also describe how a “live” event was staged when the UHD video coming from the ISS had a latency of 10+ seconds. In addition, the paper will touch on the unique collaboration between the inherently governmental aspects of the ISS, commercial partners Amazon and Elemental, and the National Association of Broadcasters.

Reinforcement learning based coding unit early termination algorithm for high efficiency video coding

In this paper, we propose a Reinforcement Learning (RL) based Coding Unit (CU) early termination algorithm for High Efficiency Video Coding (HEVC). RL is utilized to learn a CU early termination classifier independent of depths for low complexity video coding. Firstly, we model the process of CU decision as a Markov Decision Process (MDP) according to the Markov property of CU decision. Secondly, based on the MDP, a CU early termination classifier independent of depths is learned from trajectories of CU decision across different depths with the end-to-end actor-critic RL algorithm. Finally, a CU decision early termination algorithm is introduced with the learned classifier, so as to reduce computational complexity of CU decision. We implement the proposed scheme with different neural network structures. Two different neural network structures are utilized in the implementation of RL based video encoder, which are evaluated to reduce video coding complexity by 34.34% and 43.33%. With regard to Bjøntegaard delta peak signal-to-noise ratio and Bjøntegaard delta bit rate, the results are −0.033 dB and 0.85%, −0.099 dB and 2.56% respectively on average under low delay B main configuration, when compared with the HEVC test model version 16.5.

Comparing preservice teachers’ professional noticing skills in elementary mathematics classrooms

AbstractThis paper examines the implementation of an instructional module on preservice elementary teachers’ professional noticing of children’s mathematical thinking. The module focuses on professional noticing skills through the content focus of early algebraic reasoning and uses complex video vignettes from whole class instruction in authentic elementary mathematics classrooms. Findings indicated that two of the three components of professional noticing (attending and interpreting) showed statistically significant increases in a treatment group that did not occur in a comparison group. The deciding component remains a challenge that warrants further research.

End-to-End Background Subtraction via a Multi-Scale Spatio-Temporal Model

Background subtraction is an important task in computer vision. Traditional approaches usually utilize low-level visual features like color, texture, or edge to build background models. Due to the lack of deep features, they often achieve poor performance when facing complex video scenes such as illumination changes, background, or camera motions, camouflage effects and shadows. Recently, deep learning has shown to perform well in extracting deep features. To improve the robustness of background subtraction, in this paper, we propose an end-to-end multi-scale spatio-temporal (MS-ST) method which is able to extract deep features from video sequences. First, a video clip is input into a convolutional neural network for extracting multi-scale spatial features. Subsequently, to exploit the temporal information, we combine temporal sampling operations and ConvLSTM modules to extract the multi-scale temporal contextual information. Finally, the segmentation result is generated by fusing multi-scale spatio-temporal features. The experimental results on the CDnet-2014 dataset and the LASIESTA dataset demonstrate the effectiveness and superiority of the proposed method.