Monocular Video Sequences Research Articles

Moving object detection based on incremental learning low rank representation and spatial constraint

Background modeling and subtraction, the task to detect moving objects in a scene, is an important step in video analysis. In this paper, we present a novel moving object detection method based on Online Low Rank Matrix Recovery and graph cut from monocular video sequences. First, use the K-SVD method to initialize the dictionary to construct the background model, perform foreground detection with augmented Lagrange multipliers (ALM) and refine foreground values by spatial smooth constraint to extract the background and foreground information; Then obtain the clusters of foreground and background respectively using mean shift clustering on the background and foreground information; Third, initialize the S/T Network with corresponding image pixels as nodes (except S/T node); Calculate the data and smoothness term of graph; Finally, use max flow/minimum cut to segmentation S/T network to extract the motion objects. Online dictionary learning is adopted to update the background model. Experimental results on indoor and outdoor videos demonstrate the efficiency of our proposed method.

Monocular Video Guided Garment Simulation

We present a prototype to generate a garment-shape sequence guided by a monocular video sequence. It is a combination of a physically-based simulation and a boundary-based modification. Given a garment in the video worn on a mannequin, the simulation generates a garment initial shape by exploiting the mannequin shapes estimated from the video. The modification then deforms the simulated 3D shape into such a shape that matches the garment 2D boundary extracted from the video. According to the matching correspondences between the vertices on the shape and the points on the boundary, the modification is implemented by attracting the matched vertices and their neighboring vertices. For best-matching correspondences and efficient performance, three criteria are introduced to select the candidate vertices for matching. Since modifying each garment shape independently may cause inter-frame oscillations, changes by the modification are also propagated from one frame to the next frame. As a result, the generated garment 3D shape sequence is stable and similar to the garment video sequence. We demonstrate the effectiveness of our prototype with a number of examples.

Non-Rigid Structure-From-Motion on Degenerate Deformations With Low-Rank Shape Deformation Model

Non-rigid structure-from-motion (NRSfM) is the process of recovering time-varying 3D structures and poses of a deformable object from an uncalibrated monocular video sequence. Currently, most NRSfM algorithms utilize a non- degenerate assumption for non-rigid object deformations whereby the 3D structures of a non-rigid object can be assumed to be a linear combination of basis shapes with full rank three. Unfortunately, this assumption will produce extra degrees-of-freedom when the non-rigid object has some degenerate deformations with shape bases of rank less than three. These extra degrees-of-freedom will yield spurious shape deformations due to non-negligible noise in real applications, which will cause substantial reconstruction errors. To solve this problem, we propose a low-rank shape deformation model to represent 3D structures of degenerate deformations. When modeling degenerate deformations, the proposed model exploits the rank-deficient nature of degenerate deformations in addition to the low-rank property of non-rigid objects' trajectories, thus providing a more accurate and compact representation compared with existing models. Based on this model, we formulate the NRSfM problem as two coherent optimization problems. These problems are solved with iterative non-linear optimization algorithms. Experiments on synthetic and motion capture data are conducted. The results exhibit the significant advantages of our approach over state-of-the-art NRSfM algorithms for the 3D recovery of non-rigid objects with degenerate deformations.

3D Pose Tracking with Multi-Template Warping and SIFT Correspondences

Template warping is a popular technique in vision-based 3D motion tracking and 3D pose estimation due to its flexibility of being applicable to monocular video sequences. However, the method suffers from two major limitations that hamper its successful use in practice. First, it requires the camera to be calibrated prior to applying the method. Second, it may fail to provide good results if the inter-frame displacements are too large. To overcome the first problem, we propose to estimate the unknown focal length of the camera from several initial frames by an iterative optimization process. To alleviate the second problem, we propose a tracking method based on combining complementary information provided by dense optical flow and tracked scale-invariant feature transform (SIFT) features. While optical flow is good for small displacements and provides accurate local information, tracked SIFT features are better at handling larger displacements or global transformations. To combine these two pieces of complementary information, we introduce a forgetting factor to bootstrap the 3D pose estimates provided by SIFT features, and refine the final results using optical flow. Experiments are performed on three public databases, i.e., the Biwi Head Pose dataset, the BU dataset, and the McGill Faces datasets. The results illustrate that the proposed solution provides more accurate results than baseline methods that rely solely on either template warping or SIFT features. In addition, the approach can be applied in a larger variety of scenarios, due to circumventing the need for camera calibration, thus providing a more flexible solution to the problem than existing methods.

Human Action Recognition Based on 3D Human Modeling and Cyclic HMMs

Human action recognition is used in areas such as surveillance, entertainment, and healthcare. This paper proposes a system to recognize both single and continuous human actions from monocular video sequences, based on 3D human modeling and cyclic hidden Markov models (CHMMs). First, for each frame in a monocular video sequence, the 3D coordinates of joints belonging to a human object, through actions of multiple cycles, are extracted using 3D human modeling techniques. The 3D coordinates are then converted into a set of geometrical relational features (GRFs) for dimensionality reduction and discrimination increase. For further dimensionality reduction, k-means clustering is applied to the GRFs to generate clustered feature vectors. These vectors are used to train CHMMs separately for different types of actions, based on the Baum-Welch re-estimation algorithm. For recognition of continuous actions that are concatenated from several distinct types of actions, a designed graphical model is used to systematically concatenate different separately trained CHMMs. The experimental results show the effective performance of our proposed system in both single and continuous action recognition problems.

Non-Rigid Structure-From-Motion With Uniqueness Constraint and Low Rank Matrix Fitting Factorization

Non-rigid structure-from-motion is one of the difficult and challenging problems in computer vision, especially when the only input available is 2D correspondences in monocular video sequence. This paper proposed a new constraint based framework for underconstrained non-rigid structure-from-motion problem to constrain the space of solution. The proposed method is based on a point trajectory approach with an additional uniqueness constraint applied to shape coefficients to reduce the basis required to construct the non-rigid 3D shape. A framework for occluded and incomplete measured data is also proposed using low rank matrix fitting which is a robust factorization scheme for the matrix completion problem. This method offers not only new theoretical insight, but also a practical, everyday solution, to non-rigid structure-from-motion. The proposed method is positively compared to the state-of-the-art in non-rigid structure-from-motion, providing improved results on high-frequency deformations of both articulated and simpler deformable shapes.

Automatic object segmentation of unstructured scenes using colour and depth maps

This study presents a segmentation pipeline that fuses colour and depth information to automatically separate objects of interest in video sequences captured from a quadcopter. Many approaches assume that cameras are static with known position, a condition which cannot be preserved in most outdoor robotic applications. In this study, the authors compute depth information and camera positions from a monocular video sequence using structure from motion and use this information as an additional cue to colour for accurate segmentation. The authors model the problem similarly to standard segmentation routines as a Markov random field and perform the segmentation using graph cuts optimisation. Manual intervention is minimised and is only required to determine pixel seeds in the first frame which are then automatically reprojected into the remaining frames of the sequence. The authors also describe an automated method to adjust the relative weights for colour and depth according to their discriminative properties in each frame. Experimental results are presented for two video sequences captured using a quadcopter. The quality of the segmentation is compared to a ground truth and other state‐of‐the‐art methods with consistently accurate results.

Moving object detection based on improved VIBE and graph cut optimization

Extracting foreground moving objects from video sequences is an important task and also a hot topic in computer vision and image processing. Segmentation results can be used in many object-based video applications such as object-based video coding, content-based video retrieval, intelligent video surveillance and video-based human–computer interaction. In this paper, we present a novel moving object detection method based on improved VIBE and graph cut method from monocular video sequences. Firstly, perform moving object detection for the current frame based on improved VIBE method to extract the background and foreground information; then obtain the clusters of foreground and background respectively using mean shift clustering on the background and foreground information; Third, initialize the S/T Network with corresponding image pixels as nodes (except S/T node); calculate the data and smoothness term of graph; finally, use max flow/minimum cut to segmentation S/T network to extract the motion objects. Experimental results on indoor and outdoor videos demonstrate the efficiency of our proposed method.

Recovering upper-body motion using a reinitialization particle filter

An important problem associated with particle filtering is track drift, which is caused by the inaccurate likelihood measurement function and state model; another problem is the heavy computation cost in high state space. A human body is an articulated object with high degrees of freedom, and humans often perform versatile unconstrained motions. Therefore, these two problems are invariably encountered when human motion is reconstructed from monocular video sequences using particle filters. To overcome these problems, we present a novel approach to recover three-dimensional human upper-body pose using a combination of a deterministic and stochastic method that takes full advantage of the benefits of both methods—highly accurate pose reconstruction and low computation cost. The reconstruction of the human upper-body pose is divided into two parts: global pose estimation and pose estimation of the remaining joints. The global pose is determined by solving a system of six nonlinear equations established by three scale invariant feature transform (SIFT) correspondences within the left and right shoulder segments. Estimation of the pose of the remaining joints is accomplished using two particle filters, one for left arm pose estimation and the other for right arm pose estimation. The image projection of the segment model is obtained by forward kinematics under a perspective camera model, and the likelihood measurement functions involving two features are presented to enhance the model fitting performance. To avoid track drift, the particle filters can reinitialize particle sets if most of the particles deviate from the target object. The probability distribution of the new particle set is modeled by a Gaussian mixture model, in which each Gaussian is determined by matched SIFT correspondences. Experimental results show that our proposed approach effectively and efficiently achieves human upper-body tracking and is more accurate than the standard particle filter.

Monocular Human Motion Tracking by Using DE-MC Particle Filter

Tracking human motion from monocular video sequences has attracted significantly increased interests in recent years. A key to accomplishing this task is to efficiently explore a high-dimensional state space. However, the traditional particle filter method and many of its variants have not been able to meet expectations as they lack a strategy to do efficiently sampling or stochastic search. We present a novel approach, namely differential evolution-Markov chain (DE-MC) particle filtering. By taking the advantage of the DE-MC algorithm's ability to approximate complicated distributions, substantial improvement can be made to the traditional structure of the particle filter. As a result, an efficient stochastic search can be performed to locate the modes of likelihoods. Furthermore, we apply the proposed algorithm to solve the 3D articulated model-based human motion tracking problem. A reliable image likelihood function is built for visual tracker design. Based on the proposed DE-MC particle filter and the image likelihood function, we perform a variety of monocular human motion tracking experiments. Experimental results, including the comparison with the performance of other particle filtering methods demonstrate the reliable tracking performance of the proposed approach.

Hierarchical On-line Appearance-Based Tracking for 3D head pose, eyebrows, lips, eyelids and irises

In this paper, we propose an On-line Appearance-Based Tracker (OABT) for simultaneous tracking of 3D head pose, lips, eyebrows, eyelids and irises in monocular video sequences. In contrast to previously proposed tracking approaches, which deal with face and gaze tracking separately, our OABT can also be used for eyelid and iris tracking, as well as 3D head pose, lips and eyebrows facial actions tracking. Furthermore, our approach applies an on-line learning of changes in the appearance of the tracked target. Hence, the prior training of appearance models, which usually requires a large amount of labeled facial images, is avoided. Moreover, the proposed method is built upon a hierarchical combination of three OABTs, which are optimized using a Levenberg–Marquardt Algorithm (LMA) enhanced with line-search procedures. This, in turn, makes the proposed method robust to changes in lighting conditions, occlusions and translucent textures, as evidenced by our experiments. Finally, the proposed method achieves head and facial actions tracking in real-time.

Iris-Eyelid Separability Filter for Irises Tracking

This paper describes a sophisticated method to track irises in a monocular video sequence with a particle filter that uses a newly proposed Iris-Eyelid separability filter (IESF). In order to reduce the influence of eyelids and to cope with the variance of the appearance of eyes of different people, we propose an Iris-Eyelid separability filter (IESF), and use it to estimate the likelihood of hypotheses in the particle filter. We confirmed that our method has the ability of tracking various eyes for different people whose head motion including translation, rotation and zoom, even when people are wearing glasses. Through the comparative experiments of the IESF and the conventional circular separability filter (CCSF), we confirmed that our iris tracking algorithm is more robustly.

单目视频运动目标轨迹三维重建的平滑约束法

In this paper, generic smoothness constrained 3D trajectory reconstruction of moving object from monocular video is proposed. By introducing the generic smoothness constraint on the 3D trajectory, uncon- strained optimization model for 3D trajectory reconstruction is achieved and a closed-form solution is derived. Compared with the prede ned basis methods such as Discrete Cosine Transform based method and polynomial basis based method, the proposed method is more generic and can be applied to incomplete measurement case, thus having broad applicability. Geometric explanation and uniqueness of 3D trajectory reconstruction are given. Experimental results on both synthetic data and real monocular video sequences have shown the validity and advancement of the proposed method.

Variable structure multiple model for articulated human motion tracking from monocular video sequences

A new model-based human body tracking framework with learning-based theory is introduced in this paper. We propose a variable structure multiple model (VSMM) framework to address challenging problems such as uncertainty of motion styles, imprecise detection of feature points, and ambiguity of joint locations. Key human joint points are detected automatically and the undetected points are estimated with Kalman filters. Multiple motion models are learned from motion capture data using a ridge regression method. The model set that covers the total motion set is designed on the basis of topological and compatibility relationships, while the VSMM algorithm is used to estimate quaternion vectors of joint rotation. Experiments using real image sequences and simulation videos demonstrate the high efficiency of our proposed human tracking framework.

Model-based human gait tracking, 3D reconstruction and recognition in uncalibrated monocular video

Automatic analysis of human motion includes initialisation, tracking, pose recovery and activity recognition. In this paper, a computing framework is developed to automatically analyse human motions through uncalibrated monocular video sequences. A model-based kinematics approach is proposed for human gait tracking. Based on the tracking results, 3D human poses and gait features are recovered and extracted. The recognition performance is evaluated by using different classifiers. The proposed method is advantageous in its capability of recognising human subjects walking non-parallel to the image plane.

Video Normals from Colored Lights

We present an algorithm and the associated single-view capture methodology to acquire the detailed 3D shape, bends, and wrinkles of deforming surfaces. Moving 3D data has been difficult to obtain by methods that rely on known surface features, structured light, or silhouettes. Multispectral photometric stereo is an attractive alternative because it can recover a dense normal field from an untextured surface. We show how to capture such data, which in turn allows us to demonstrate the strengths and limitations of our simple frame-to-frame registration over time. Experiments were performed on monocular video sequences of untextured cloth and faces with and without white makeup. Subjects were filmed under spatially separated red, green, and blue lights. Our first finding is that the color photometric stereo setup is able to produce smoothly varying per-frame reconstructions with high detail. Second, when these 3D reconstructions are augmented with 2D tracking results, one can register both the surfaces and relax the homogenous-color restriction of the single-hue subject. Quantitative and qualitative experiments explore both the practicality and limitations of this simple multispectral capture system.

A Fast Bilinear Structure from Motion Algorithm Using a Video Sequence and Inertial Sensors

In this paper, we study the benefits of the availability of a specific form of additional information—the vertical direction (gravity) and the height of the camera, both of which can be conveniently measured using inertial sensors and a monocular video sequence for 3D urban modeling. We show that in the presence of this information, the SfM equations can be rewritten in a bilinear form. This allows us to derive a fast, robust, and scalable SfM algorithm for large scale applications. The SfM algorithm developed in this paper is experimentally demonstrated to have favorable properties compared to the sparse bundle adjustment algorithm. We provide experimental evidence indicating that the proposed algorithm converges in many cases to solutions with lower error than state-of-art implementations of bundle adjustment. We also demonstrate that for the case of large reconstruction problems, the proposed algorithm takes lesser time to reach its solution compared to bundle adjustment. We also present SfM results using our algorithm on the Google StreetView research data set.

VideoMocap

This paper presents a video-based motion modeling technique for capturing physically realistic human motion from monocular video sequences. We formulate the video-based motion modeling process in an image-based keyframe animation framework. The system first computes camera parameters, human skeletal size, and a small number of 3D key poses from video and then uses 2D image measurements at intermediate frames to automatically calculate the "in between" poses. During reconstruction, we leverage Newtonian physics, contact constraints, and 2D image measurements to simultaneously reconstruct full-body poses, joint torques, and contact forces. We have demonstrated the power and effectiveness of our system by generating a wide variety of physically realistic human actions from uncalibrated monocular video sequences such as sports video footage.

VideoMocap

This paper presents a video-based motion modeling technique for capturing physically realistic human motion from monocular video sequences. We formulate the video-based motion modeling process in an image-based keyframe animation framework. The system first computes camera parameters, human skeletal size, and a small number of 3D key poses from video and then uses 2D image measurements at intermediate frames to automatically calculate the "in between" poses. During reconstruction, we leverage Newtonian physics, contact constraints, and 2D image measurements to simultaneously reconstruct full-body poses, joint torques, and contact forces. We have demonstrated the power and effectiveness of our system by generating a wide variety of physically realistic human actions from uncalibrated monocular video sequences such as sports video footage.

Articulated body motion tracking using illumination invariant optical flow

We propose a model-based tracking method for articulated objects in monocular video sequences under varying illumination conditions. The tracking method uses estimates of optical flows constructed by projecting model textures into the camera images and comparing the projected textures with the recorded information. An articulated body is modelled in terms of 3D primitives, each possessing a specified texture on its surface. An important step in model-based tracking of 3D objects is the estimation of the pose of the object during the tracking process. The optimal pose is estimated by minimizing errors between the computed optical flow and the projected 2D velocities of the model textures. This estimation uses a least-squares method with kinematic constraints for the articulated object and a perspective camera model. We test our framework with an articulated robot and show results.