Dynamic Texture Model Research Articles

Kernelized Similarity Learning and Embedding for Dynamic Texture Synthesis

Dynamic texture (DT) exhibits statistical stationarity in the spatial domain and stochastic repetitiveness in the temporal dimension, indicating that different frames of DT possess a high similarity correlation that is critical prior knowledge. However, existing methods cannot effectively learn a synthesis model for high-dimensional DT from a small number of training samples. In this article, we propose a novel DT synthesis method, which makes full use of similarity as prior knowledge to address this issue. Our method is based on the proposed kernel similarity embedding, which can not only mitigate the high dimensionality and small sample issues, but also has the advantage of modeling nonlinear feature relationships. Specifically, we first put forward two hypotheses that are essential for the DT model to generate new frames using similarity correlations. Then, we integrate kernel learning and the extreme learning machine into a unified synthesis model to learn kernel similarity embeddings for representing DTs. Extensive experiments on DT videos collected from the Internet and two benchmark datasets, i.e., Gatech Graphcut Textures and Dyntex, demonstrate that the learned kernel similarity embeddings can provide discriminative representations for DTs. Further, our method can preserve the long-term temporal continuity of the synthesized DT sequences with excellent sustainability and generalization. Meanwhile, it effectively generates realistic DT videos with higher speed and lower computation than the current state-of-the-art methods. The code and more synthesis videos are available at our project page <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://shiming-chen.github.io/Similarity-page/Similarit.html</uri> .

Real-time deep dynamic characters

We propose a deep videorealistic 3D human character model displaying highly realistic shape, motion, and dynamic appearance learned in a new weakly supervised way from multi-view imagery. In contrast to previous work, our controllable 3D character displays dynamics, e.g., the swing of the skirt, dependent on skeletal body motion in an efficient data-driven way, without requiring complex physics simulation. Our character model also features a learned dynamic texture model that accounts for photo-realistic motion-dependent appearance details, as well as view-dependent lighting effects. During training, we do not need to resort to difficult dynamic 3D capture of the human; instead we can train our model entirely from multi-view video in a weakly supervised manner. To this end, we propose a parametric and differentiable character representation which allows us to model coarse and fine dynamic deformations, e.g., garment wrinkles, as explicit spacetime coherent mesh geometry that is augmented with high-quality dynamic textures dependent on motion and view point. As input to the model, only an arbitrary 3D skeleton motion is required, making it directly compatible with the established 3D animation pipeline. We use a novel graph convolutional network architecture to enable motion-dependent deformation learning of body and clothing, including dynamics, and a neural generative dynamic texture model creates corresponding dynamic texture maps. We show that by merely providing new skeletal motions, our model creates motion-dependent surface deformations, physically plausible dynamic clothing deformations, as well as video-realistic surface textures at a much higher level of detail than previous state of the art approaches, and even in real-time.

Real-time deep dynamic characters

We propose a deep videorealistic 3D human character model displaying highly realistic shape, motion, and dynamic appearance learned in a new weakly supervised way from multi-view imagery. In contrast to previous work, our controllable 3D character displays dynamics, e.g., the swing of the skirt, dependent on skeletal body motion in an efficient data-driven way, without requiring complex physics simulation. Our character model also features a learned dynamic texture model that accounts for photo-realistic motion-dependent appearance details, as well as view-dependent lighting effects. During training, we do not need to resort to difficult dynamic 3D capture of the human; instead we can train our model entirely from multi-view video in a weakly supervised manner. To this end, we propose a parametric and differentiable character representation which allows us to model coarse and fine dynamic deformations, e.g., garment wrinkles, as explicit spacetime coherent mesh geometry that is augmented with high-quality dynamic textures dependent on motion and view point. As input to the model, only an arbitrary 3D skeleton motion is required, making it directly compatible with the established 3D animation pipeline. We use a novel graph convolutional network architecture to enable motion-dependent deformation learning of body and clothing, including dynamics, and a neural generative dynamic texture model creates corresponding dynamic texture maps. We show that by merely providing new skeletal motions, our model creates motion-dependent surface deformations, physically plausible dynamic clothing deformations, as well as video-realistic surface textures at a much higher level of detail than previous state of the art approaches, and even in real-time.

Low-Cost Approach for Improving Video Transmission Efficiency in WVSN

The wireless visual sensor network (WVSN) as a new emerged intelligent visual system, has been applied in many video monitoring sites. However, there is still great challenge because of the limited wireless network bandwidth. To resolve the problem, we propose a real-time dynamic texture approach which can detect and reduce the temporal redundancy during many successive image frames. Firstly, an adaptively learning background model is improved to discover successive similar image frames from the inputting video sequence. Then, the dynamic texture model based on the singular value decomposition is adopted to distinguish foreground and background element dynamics. Furthermore, a background discarding strategy based on visual motion coherence is proposed to determine whether each image frame is streamed or not. To evaluate the trade-off performance of the proposed method, it is tested on the CDW-2014 dataset, which can accurately detect the first foreground frame when the moving objects of interest appear in the field of view in the most tested dynamic scenes, and the misdetection rate of the undetected foreground frames is near to zero. Compared to the original stream, it can reduce the occupied bandwidth a lot and its computational cost is relatively lower than the state-of-the-art methods.

Bayesian Modeling of Motion Perception Using Dynamical Stochastic Textures.

A common practice to account for psychophysical biases in vision is to frame them as consequences of a dynamic process relying on optimal inference with respect to a generative model. The study presented here details the complete formulation of such a generative model intended to probe visual motion perception with a dynamic texture model. It is derived in a set of axiomatic steps constrained by biological plausibility. We extend previous contributions by detailing three equivalent formulations of this texture model. First, the composite dynamic textures are constructed by the random aggregation of warped patterns, which can be viewed as three-dimensional gaussian fields. Second, these textures are cast as solutions to a stochastic partial differential equation (sPDE). This essential step enables real-time, on-the-fly texture synthesis using time-discretized autoregressive processes. It also allows for the derivation of a local motion-energy model, which corresponds to the log likelihood of the probability density. The log likelihoods are essential for the construction of a Bayesian inference framework. We use the dynamic texture model to psychophysically probe speed perception in humans using zoom-like changes in the spatial frequency content of the stimulus. The human data replicate previous findings showing perceived speed to be positively biased by spatial frequency increments. A Bayesian observer who combines a gaussian likelihood centered at the true speed and a spatial frequency dependent width with a "slow-speed prior" successfully accounts for the perceptual bias. More precisely, the bias arises from a decrease in the observer's likelihood width estimated from the experiments as the spatial frequency increases. Such a trend is compatible with the trend of the dynamic texture likelihood width.

Classifying terrain properties for planetary exploration rovers based on a combined distance measure with dynamic texture model

Non-geometric terrain properties surrounding a planetary exploration rover can be exploited to improve autonomous mobility of the rover. This paper shows a method to classify non-geometric terrain properties using image sequences obtained from onboard cameras. Our method is based on a Dynamic Texture analysis, which is a technique to estimate scene motion in image sequences. Using Dynamic Textures, we can incorporate a motion cue to classify not only soil types but also the velocities of a rover relative to terrain surface representing slippage due to terrains. First, we briefly show a learning algorithm for Dynamic Textures. Then we propose a combined distance measure for classification. Combining distance measures is useful to handle different properties of terrain, that is, a static property such as soil types and a dynamic property such as the velocities. The effectiveness of the combined distance measure for improving classification performance is demonstrated through experimental runs of two rover testbeds on sand pits.

Synthesizing and editing dynamic flames with color temperature

Generating computer animated flames is a difficult and computationally expensive problem. Dynamic textures provide an effective means for extrapolating and synthesizing dynamic flames, but aggravates color distortion due to the high correlation of RGB components. A novel method for dynamic flame texture synthesis using color temperature is proposed in this paper. Firstly, the color-temperature mapping is calculated by using the Planck’s law and two-color pyrometric technique to avoid color distortion. Secondly, a novel dynamic texture model is presented to transform the RGB space into temperature space. Finally, the dynamic flames editing is presented to support physical temperature adjustment. Experimental results illustrate that our approach is effective to synthesize visually plausible dynamic flames without color distortions and to edit dynamic flames with intuitive physical interpretation.

SPATIAL MOTION PATTERNS: ACTION MODELS FROM SEMI-DENSE TRAJECTORIES

A new action model is proposed, by revisiting local binary patterns (LBP) for dynamic texture models, applied on trajectory beams calculated on the video. The use of semi-dense trajectory field allows to dramatically reduce the computation support to essential motion information, while maintaining a large amount of data to ensure robustness of statistical bag of features action models. A new binary pattern, called Spatial Motion Pattern (SMP) is proposed, which captures self-similarity of velocity around each tracked point (particle), along its trajectory. This operator highlights the geometric shape of rigid parts of moving objects in a video sequence. SMPs are combined with basic velocity information to form the local action primitives. Then, a global representation of a space × time video block is provided by using hierarchical blockwise histograms, which allows to efficiently represent the action as a whole, while preserving a certain level of spatiotemporal relation between the action primitives. Inheriting from the efficiency and the invariance properties of both the semi-dense tracker Video extruder and the LBP-based representations, the method is designed for the fast computation of action descriptors in unconstrained videos. For improving both robustness and computation time in the case of high definition video, we also present an enhanced version of the semi-dense tracker based on the so-called super particles, which reduces the number of trajectories while improving their length, reliability and spatial distribution.

Complex background subtraction by pursuing dynamic spatio-temporal models.

Although it has been widely discussed in video surveillance, background subtraction is still an open problem in the context of complex scenarios, e.g., dynamic backgrounds, illumination variations, and indistinct foreground objects. To address these challenges, we propose an effective background subtraction method by learning and maintaining an array of dynamic texture models within the spatio-temporal representations. At any location of the scene, we extract a sequence of regular video bricks, i.e., video volumes spanning over both spatial and temporal domain. The background modeling is thus posed as pursuing subspaces within the video bricks while adapting the scene variations. For each sequence of video bricks, we pursue the subspace by employing the auto regressive moving average model that jointly characterizes the appearance consistency and temporal coherence of the observations. During online processing, we incrementally update the subspaces to cope with disturbances from foreground objects and scene changes. In the experiments, we validate the proposed method in several complex scenarios, and show superior performances over other state-of-the-art approaches of background subtraction. The empirical studies of parameter setting and component analysis are presented as well.

Clustering Dynamic Textures with the Hierarchical EM Algorithm for Modeling Video

Dynamic texture (DT) is a probabilistic generative model, defined over space and time, that represents a video as the output of a linear dynamical system (LDS). The DT model has been applied to a wide variety of computer vision problems, such as motion segmentation, motion classification, and video registration. In this paper, we derive a new algorithm for clustering DT models that is based on the hierarchical EM algorithm. The proposed clustering algorithm is capable of both clustering DTs and learning novel DT cluster centers that are representative of the cluster members in a manner that is consistent with the underlying generative probabilistic model of the DT. We also derive an efficient recursive algorithm for sensitivity analysis of the discrete-time Kalman smoothing filter, which is used as the basis for computing expectations in the E-step of the HEM algorithm. Finally, we demonstrate the efficacy of the clustering algorithm on several applications in motion analysis, including hierarchical motion clustering, semantic motion annotation, and learning bag-of-systems (BoS) codebooks for dynamic texture recognition.

Estimating the Video Registration Using Image Motions

In this research, we consider the problems of registering multiple video sequences dynamic scenes which are not limited non rigid objects such as fireworks, blasting, high speed car moving taken from different vantage points. In this paper we propose a simple algorithm we can create different frames on particular videos moving for matching such complex scenes. Our algorithm does not require the cameras to be synchronized, and is not based on frame-by-frame or volume-by-volume registration. Instead, we model each video as the output of a linear dynamical system and transform the task of registering the video sequences to that of registering the parameters of the corresponding dynamical models. In this paper we use of a joint frame together to form distinct frame concurrently. The joint identification and the Jordan canonical form are not only applicable to the case of registering video sequences, but also to the entire genre of algorithms based on the dynamic texture model. We have also shown that out of all the possible choices for the method of identification and canonical form, the JID using JCF performs the best.

Intrackability: Characterizing Video Statistics and Pursuing Video Representations

Videos of natural environments contain a wide variety of motion patterns of varying complexities which are represented by many different models in the vision literature. In many situations, a tracking algorithm is formulated as maximizing a posterior probability. In this paper, we propose to measure the video complexity by the entropy of the posterior probability, called the intrackability, to characterize the video statistics and pursue optimal video representations. Based on the definition of intrackability, our study is aimed at three objectives. Firstly, we characterize video clips of natural scenes by intrackability. We calculate the intrackabilities of image points to measure the local inferential uncertainty, and collect the histogram of the intrackabilities over the video in space and time as the global video statistics. We find that a PCA scatter-plot based on the first two principle components of intrackability histograms can reflect the major variations, i.e., image scaling and object density, in natural video clips. Secondly, we show that different video representations, including deformable contours, tracking kernels with various appearance features, dense motion fields, and dynamic texture models, are connected by the change of intrackability and thus develop a simple criterion for model transition and for pursuing the optimal video representation. Thirdly, we derive the connections between the intrackability measure and other criteria in the literature such as the Shi-Tomasi texturedness measure, conditional number, and Harris-Stephens R score, and compare with the Shi-Tomasi measure in tracking experiments.

Gait recognition based on improved dynamic Bayesian networks

In this paper, we proposed an improved two-level dynamic Bayesian network layered time series model (LTSM), which aims to solve the limitations hindering the application of available dynamic Bayesian networks, the hidden Markov model (HMM) and the dynamic texture (DT) model to gait recognition. In the first level, a gait silhouette or feature cycle is divided into several temporally adjacent clusters. Each cluster is modeled by a DT or logistic DT (LDT). In the second level, HMM is built to describe the relationship among the DTs/LDTs. Besides LTSM, LDT is also an improved dynamic Bayesian network presented in this paper to describe the binary image sequence, which introduces the logistic principle component analysis (PCA) to learning its parameters. We demonstrated the validity of LTSM with experiments on both the CMU Mobo gait database and CASIA gait database (dataset B), and that of LDT on the CMU Mobo gait database. Experimental results showed the superiority of the improved dynamic Bayesian networks.

Modeling Music as a Dynamic Texture

We consider representing a short temporal fragment of musical audio as a <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">dynamic texture</i> , a model of both the timbral and rhythmical qualities of sound, two of the important aspects required for automatic music analysis. The dynamic texture model treats a <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">sequence</i> of audio feature vectors as a sample from a linear dynamical system. We apply this new representation to the task of automatic song segmentation. In particular, we cluster audio fragments, extracted from a song, as samples from a dynamic texture mixture (DTM) model. We show that the DTM model can both accurately cluster coherent segments in music and detect transition boundaries. Moreover, the generative character of the proposed model of music makes it amenable for a wide range of applications besides segmentation. As examples, we use DTM models of songs to suggest possible improvements in other music information retrieval applications such as music annotation and similarity.

Layered time series model for gait recognition

A new gait recognition algorithm, the layered time series model (LTSM), is proposed. LTSM is a two-level model which combines the dynamic texture model (DTM) and the hidden Markov model (HMM). A gait cycle is divided into several temporally adjacent clusters and gait features of each cluster are modelled by the DTM. The HMM is built to describe the relationship among the DTMs, which are regarded as hidden states. Experiment results show that the proposed model outperforms other approaches in terms of recognition accuracy.