Pair Of Frames Research Articles

Volumetric spatial feature representation for view-invariant human action recognition using a depth camera

The problem of viewpoint variations is a challenging issue in vision-based human action recognition. With the richer information provided by three-dimensional (3-D) point clouds thanks to the advent of 3-D depth cameras, we can effectively analyze spatial variations in human actions. In this paper, we propose a volumetric spatial feature representation (VSFR) that measures the density of 3-D point clouds for view-invariant human action recognition from depth sequence images. Using VSFR, we construct a self-similarity matrix (SSM) that can graphically represent temporal variations in the depth sequence. To obtain an SSM, we compute the squared Euclidean distance of VSFRs between a pair of frames in a video sequence. In this manner, an SSM represents the dissimilarity between a pair of frames in terms of spatial information in a video sequence captured at an arbitrary viewpoint. Furthermore, due to the use of a bag-of-features method for feature repre- sentations, the proposed method efficiently handles the variations of action speed or length. Hence, our method is robust to both variations in viewpoints and lengths of action sequences. We evaluated the proposed method by comparing with state-of-the-art methods in the literature on three public datasets of ACT4 2 , MSRAction3D, and MSRDailyActivity3D, validating the superiority of our method by achieving the highest accuracies. © 2015 Society of

A near lossless compression domain volume rendering algorithm for floating-point time-varying volume data

Compressing floating-point time-varying volume data and achieving both high compression rate and near lossless are challenging. This paper proposes a compression domain volume rendering (CDVR) approach based on hierarchical vector quantization (HVQ) and perfect spatial hashing (PSH) techniques. First, a HVQ process is applied to the first frame to obtain codebooks and index volumes. Then, a sparse residual volume (SRV) is computed by differencing the first frame and the recovery volume, which is reconstructed by utilizing the codebooks and the index volumes. Difference volumes are calculated by differencing the adjacent frame pairs of the time-series. Thereafter, both the SRV and the difference volumes are compressed by means of PSH technique. To render the time-series, the codebooks, the index volumes and the results of PSH are decompressed on-the-fly in constant time in GPU. In addition, a high compression rate is achieved by HVQ and PSH, and the compression is near lossless. The results on varied datasets verify that the proposed method can achieve the high compression rate and near lossless compression quality for floating-point time-varying volume data, as well as high efficient CDVR.

Pairs of dual Gabor frames generated by functions of Hilbert-Schmidt type

We show that any two functions which are real-valued, bounded, compactly supported and whose integer translates each form a partition of unity lead to a pair of windows generating dual Gabor frames for L2(?)$L^{2}(\mathbb {R})$. In particular we show that any such functions have families of dual windows where each member may be written as a linear combination of integer translates of any B-spline. We introduce functions of Hilbert-Schmidt type along with a new method which allows us to associate to certain such functions finite families of recursively defined dual windows of arbitrary smoothness. As a special case we show that any exponential B-spline has finite families of dual windows, where each member may be conveniently written as a linear combination of another exponential B-spline. Unlike results known from the literature we avoid the usual need for the partition of unity constraint in this case.

Using Neural Networks for Determining Velocity Vectors of Air Flow Visualized by Helium Bubbles

One of the important characteristics of air flow is the velocity of flow. To determine the speed, in addition to other methods, we also use the helium bubbles seeding visualization method, when bubbles are injected into the air stream so that the air flow is obvious to the eye. If a video of such flow is taken, it is possible to determine velocity vectors in a pair of consecutive frames of this video footage, derived from the change in position of individual bubbles.This article describes a method of monitoring the bubbles in consecutive video frames. During this process, helium bubbles are detected in the first image of the pair, either by a firmly defined procedure, or with the use of a neural network. For detected bubbles, the velocity vectors are determined according to the way in which they move, therefore, according to their location in the following frame. Another neural network then determines the velocity vector at any point of image, which will be implemented in the construction of vector maps for the first image. A vector map is used for comprehensive evaluation of air flow and thus, plays an important role in the development of ventilation and air conditioning systems.

Locus equations estimated form a corpus of running speech

Locus equations, or the linear relationship between onset and vowel second-formant frequency F2 in terms of slope and y-intercept, have been presented as possible invariant correlates to consonant place of articulation [e.g., Sussman et al. (1998). Behav. Brain Sci. 21, 241–299]. In the current study, formant measurements were extracted from both stressed and unstressed vowels taken from a database of spontaneous and read speech. Locus equations were estimated for several places of articulation of the preceding consonant. In addition, optimal time frames for estimating locus equations are determined with reference to automatic classification of consonant place of articulation as well as vowel identification. Formant frequencies are first measured at multiple time frames—both before and after voicing onset in the case of voiceless plosives—to find the pair of time frames that best estimates place of articulation via discriminant analysis and other classification methods. In addition, locus-equation slopes are compared between stressed and unstressed vowels as well as between spontaneous and read speech samples. In addition, the role of total vowel duration across these contexts is described. The evaluation of several strategies for optimizing the automatic extraction of formant frequencies from running speech are also reported. [Work supported by SSHRC.]

Video tonal stabilization via color states smoothing.

We address the problem of removing video color tone jitter that is common in amateur videos recorded with hand-held devices. To achieve this, we introduce color state to represent the exposure and white balance state of a frame. The color state of each frame can be computed by accumulating the color transformations of neighboring frame pairs. Then, the tonal changes of the video can be represented by a time-varying trajectory in color state space. To remove the tone jitter, we smooth the original color state trajectory by solving an L1 optimization problem with PCA dimensionality reduction. In addition, we propose a novel selective strategy to remove small tone jitter while retaining extreme exposure and white balance changes to avoid serious artifacts. Quantitative evaluation and visual comparison with previous work demonstrate the effectiveness of our tonal stabilization method. This system can also be used as a preprocessing tool for other video editing methods.

Augmenting analytic SFM filters with frame-to-frame features

In Structure From Motion (SFM), image features are matched in either an extended number of frames or only in pairs of consecutive frames. Traditionally, SFM filters have been applied using only one of the two matching paradigms, with the Long Range (LR) feature technique being more popular because of the fact that features that are matched across multiple frames provide stronger constraints on structure and motion. Nevertheless, Frame-to-Frame (F2F) features possess the desirable property of being abundant because of the large similarity that exists between closely spaced frames. Although the use of such features has been limited mostly to the determination of inter-frame camera motion, we argue that significant improvements can be attained in online filter-based SFM by integrating the F2F features into filters that use LR features. The main contributions of this paper are twofold. First, it presents a new method that enables the incorporation of F2F information in any analytical filter in a fashion that requires minimal change to the existing filter. Our results show that by doing so, large increases in accuracy are achieved in both the structure and motion estimates. Second, thanks to mathematical simplifications we realize in the filter, we minimize the computational burden of F2F integration by two orders of magnitude, thereby enabling its real-time implementation. Experimental results on real and simulated data prove the success of the proposed approach.

Classification of Cinematographic Shots Using Lie Algebra and its Application to Complex Event Recognition

In this paper, we propose a discriminative representation of a video shot based on its camera motion and demonstrate how the representation can be used for high level multimedia tasks like complex event recognition. In our technique, we assume that a homography exists between a pair of subsequent frames in a given shot. Using purely image-based methods, we compute homography parameters that serve as coarse indicators of the ambient camera motion. Next, using Lie algebra, we map the homography matrices to an intermediate vector space that preserves the intrinsic geometric structure of the transformation. The mappings are stacked temporally to generate vector time-series per shot. To extract meaningful features from time-series, we propose an efficient linear dynamical system based technique. The extracted temporal features are further used to train linear SVMs as classifiers for a particular shot class. In addition to demonstrating the efficacy of our method on a novel dataset, we extend its applicability to recognize complex events in large scale videos under unconstrained scenarios. Our empirical evaluations on eight cinematographic shot classes show that our technique performs close to approaches that involve extraction of 3-D trajectories using computationally prohibitive structure from motion techniques.

Critical Pairs of Sequences of a Mixed Frame Potential

The classical frame potential in a finite-dimensional Hilbert space has been introduced by Benedetto and Fickus, who showed that all finite unit-norm tight frames can be characterized as the minimizers of this energy functional. This was the starting point of a series of new results in frame theory, related to finding tight frames with determined lengths. The frame potential has been studied in the traditional setting as well as in the finite-dimensional fusion frame context. In this work we introduce the concept of mixed frame potential, which generalizes the notion of the Benedetto-Fickus frame potential. We study properties of this new potential, and give the structure of its critical pairs of sequences on a suitable restricted domain. For a given sequence {α m } m=1,…, N in K, where K is ℝ or ℂ, we obtain necessary and sufficient conditions in order to have a dual pair of frames {f m } m=1,…, N , {g m } m=1,…, N such that ⟨ f m , g m ⟩ = α m for all m = 1,…, N.

On entire functions restricted to intervals, partition of unities, and dual Gabor frames

Partition of unities appears in many places in analysis. Typically it is generated by compactly supported functions with a certain regularity. In this paper we consider partition of unities obtained as integer-translates of entire functions restricted to finite intervals. We characterize the entire functions that lead to a partition of unity in this way, and we provide characterizations of the “cut-off” entire functions, considered as functions of a real variable, to have desired regularity. In particular we obtain partition of unities generated by functions with small support and desired regularity. Applied to Gabor analysis this leads to constructions of dual pairs of Gabor frames with low redundancy, generated by trigonometric polynomials with small support and desired regularity.

Detection of highly articulated moving objects by using co-segmentation with application to athletic video sequences

In this paper, we adapt the co-segmentation for the fundamental problem of segmenting automatically and accurately highly articulated athletes in a large variety of poses without any initialization or prior knowledge. Our intention is to reduce the complexity of athlete segmentation by formulating it as a constrained 2D pair of frames’ co-segmentation, in order to extract the common foreground objects under unconstrained environments without any user input. In fact, the co-segmentation allows to integrate implicitly the temporal information for automatic moving object segmentation without any assumption or prior knowledge on camera motion. The proposed method was applied on various real-world video sequences of athletic sports meetings, and promising results are obtained. Experiments show that suggested method witnessed a significant improvement over background subtraction methods, which are commonly used for athlete segmentation.

A PAIR OF ORTHOGONAL WAVELET FRAMES IN L2(ℝd)

A simple method of construction of a pair of orthogonal wavelet frames in L2(ℝd) is presented. This is a generalization of one-dimensional case to higher dimension. The construction is based on the well-known Unitary Extension Principle (UEP). The presented method produces the polyphase components of the filters of the wavelet functions, and hence the filters. A pair of orthogonal wavelet frames can be constructed with an extra condition. In the construction, the polyphase matrix is used as opposed to the modulation matrix. This is less restrictive and yields a fewer wavelet functions in the system than in the previously known constructions.

Syntax-driven semantic frame composition in Lexicalized Tree Adjoining Grammars

The grammar framework presented in this paper combines Lexicalized Tree Adjoining Grammar (LTAG) with a (de)compositional frame semantics. We introduce elementary constructions as pairs of elementary LTAG trees and decompositional frames. The linking between syntax and semantics can largely be captured by such constructions since in LTAG, elementary trees represent full argument projections. Substitution and adjunction in the syntax then trigger the unification of the associated semantic frames, which are formally defined as base-labelled feature structures. Moreover, the system of elementary constructions is specified in a metagrammar by means of tree and frame descriptions. This metagrammatical factorization gives rise to a fine-grained decomposition of the semantic contributions of syntactic building blocks, and it allows us to separate lexical from constructional contributions and to carve out generalizations across constructions. In the second half of the paper, we apply the framework to the analysis of directed motion expressions and of the dative alternation in English, two well-known examples of the interaction between lexical and constructional meaning.

Structure of nonstationary Gabor frames and their dual systems

We investigate the structural properties of dual systems for nonstationary Gabor frames. In particular, we prove that some inverse nonstationary Gabor frame operators admit a Walnut-like representation, i.e. the operator acting on a function can be described by weighted translates of that function. In this case, which only occurs when compactly supported window functions are used, the canonical dual frame partially inherits the structure of the original frame, with differences that we describe in detail. Moreover, we determine a necessary and sufficient condition for a pair of nonstationary Gabor frames to form dual frames, valid under mild restrictions. This condition is then applied in a simple setup, to prove the existence of dual pairs of nonstationary Gabor systems with coarser frequency sampling than allowed by previous results [3]. We also explore a connection to recent work of Christensen, Kim and Kim on Gabor frames with compactly supported window function.

SIFT and SURF Performance Evaluation for Mobile Robot-Monocular Visual Odometry

Visual odometry is the process of estimating the motion of mobile through the camera attached to it, by matching point features between pairs of consecutive image frames. For mobile robots, a reliable method for comparing images can constitute a key component for localization and motion estimation tasks. In this paper, we study and compare the SIFT and SURF detector/ descriptor in terms of accurate motion determination and runtime efficiency in context the mobile robot-monocular visual odometry. We evaluate the performance of these detectors/ descriptors from the repeatability, recall, precision and cost of computation. To estimate the relative pose of camera from outlier-contaminated feature correspondences, the essential matrix and inlier set is estimated using RANSAC. Experimental results demonstrate that SURF, outperform the SIFT, in both accuracy and speed. 

Island analysis of low-activity dynamic speckles

In this work we present a method to evaluate activity in low dynamic speckle patterns. It consists of binarizing the speckle image and analyzing the displacements and deformations of the resulting speckle grain regions, here called islands. Numerical simulations and controlled experiments were used to study the variations of the island features with the aim of finding a correlation with the activity of the speckle pattern. From the obtained results it was possible to conclude that the developed method can be useful for the analysis of low activity speckle patterns with the advantage of requiring only pairs of frames, thus permitting the assessment of nonstationary processes. In the case of stationary phenomena, so that stacks of frames registers are representative of them, dilute activity images can also be constructed.

Triple Epstein Frame First-Level Weighted Processing Method for Determining the Effective Path Length of the Epstein Frame

Based on double Epstein frame measurement method, double Epstein frame measurement experiments are carried out for 30P120 silicon steel samples using two pairs of Epstein frames (namely 2E (25-20) and 2E (20-17.5)) magnetic measuring devices, to study factors affecting the mean path length of the Epstein frame. 3D simulation proves the preconditions of double Epstein frame method, that is equivalent magnetic flux density of corresponding corner positions. On this basis, a weighted processing method is proposed to determine the effective path length of standard 25cm Epstein frame and to compare with that of double Epstein frame method, which processes the mean path length determined by the different specific losses of the Epstein frame (i.e., the uniform areas, the corner areas and the impact zones between them of the whole Epstein frame). Not only the method excludes the effect of the magnetic characteristics inhomogeneity of samples in the corner double-lapped structure areas, but also considers the influence of the specific power losses in the different parts determining the mean path length. The obtained experimental results confirm the correctness of the method in this paper for determining the specific total loss of electrical steel sheet.

Action recognition from a different view

In this paper, we propose a novel approach to recognise human activities from a different view. Although appearance-based recognition methods have been shown to be unsuitable for action recognition for varying views, there must be some regularity among the same action sequences of different views. Self-similarity matrices appear to be relative stable across views. However, the ability to effectively realise this stability is a problem. In this paper, we extract the shape-flow descriptor as the low-level feature and then choose the same number of key frames from the action sequences. Self-similarity matrices are obtained by computing the similarity between any pair of the key frames. The diagonal features of the similarity matrices are extracted as the highlevel feature representation of the action sequence and Support Vector Machines (SVM) is employed for classification. We test our approach on the IXMAS multi-view data set. The proposed approach is simple but effective when compared with other algorithms.

Learning to Produce 3D Media From a Captured 2D Video

Due to the advances in display technologies and commercial success of 3D motion pictures in recent years, there is renewed interest in enabling consumers to create 3D content. While new 3D content can be created using more advanced capture devices (i.e., stereo cameras), most people still own 2D capture devices. Further, enormously large collections of captured media exist only in 2D. We present a system for producing pseudo-stereo images from captured 2D videos. Our system employs a two-phase procedure where the first phase detects “good” pseudo-stereo images frames from a 2D video, which was captured a priori without any constraints on camera motion or content. We use a trained classifier to detect pairs of video frames that are suitable for constructing pseudo-stereo images. In particular, for a given frame at time t, we determine if exists such that It+t and It can form an acceptable pseudo-stereo image. Moreover, even if t is determined, generating a good pseudo-stereo image from 2D captured video frames can be nontrivial since in many videos, professional or amateur, both foreground and background objects may undergo complex motion. Independent foreground motions from different scene objects define different epipolar geometries that cause the conventional method of generating pseudo-stereo images to fail. To address this problem, the second phase of the proposed system further recomposes the frame pairs to ensure consistent 3D perception for objects for such cases. In this phase, final left and right pseudo-stereo images are created by recompositing different regions of the initial frame pairs to ensure a consistent camera geometry. We verify the performance of our method for producing pseudo-stereo media from captured 2D videos in a psychovisual evaluation using both professional movie clips and amateur home videos.

New inequalities on sparse representation in pairs of bases

In this study, the authors investigated some new inequalities on sparse representation for pairs of bases and frames, which would enrich the theory ensemble. First, for fixed pairs of bases, frames and the signal to be represented, we presented the bounds (which can be used in practice directly) of l 0-norm of signal coefficients by the max/min cross-inner-products, as is of much importance to bases and frames selections in sparse representation. Also, the error bounds associated with the given min cross-inner-products are achieved. Moreover, the equivalence condition between l 0 solution and l 1 solution was achieved via relations of cross-inner-products, as can be applied directly for selection of optimal bases. Finally, the estimation of the parameters of cross-inner-products was shown.