3D Object Tracking Research Articles

Tracking Motion, Deformation, and Texture Using Conditionally Gaussian Processes

We present a generative model and inference algorithm for 3D nonrigid object tracking. The model, which we call G-flow, enables the joint inference of 3D position, orientation, and nonrigid deformations, as well as object texture and background texture. Optimal inference under G-flow reduces to a conditionally Gaussian stochastic filtering problem. The optimal solution to this problem reveals a new space of computer vision algorithms, of which classic approaches such as optic flow and template matching are special cases that are optimal only under special circumstances. We evaluate G-flow on the problem of tracking facial expressions and head motion in 3D from single-camera video. Previously, the lack of realistic video data with ground truth nonrigid position information has hampered the rigorous evaluation of nonrigid tracking. We introduce a practical method of obtaining such ground truth data and present a new face video data set that was created using this technique. Results on this data set show that G-flow is much more robust and accurate than current deterministic optic-flow-based approaches.

Articulated body motion tracking using illumination invariant optical flow

We propose a model-based tracking method for articulated objects in monocular video sequences under varying illumination conditions. The tracking method uses estimates of optical flows constructed by projecting model textures into the camera images and comparing the projected textures with the recorded information. An articulated body is modelled in terms of 3D primitives, each possessing a specified texture on its surface. An important step in model-based tracking of 3D objects is the estimation of the pose of the object during the tracking process. The optimal pose is estimated by minimizing errors between the computed optical flow and the projected 2D velocities of the model textures. This estimation uses a least-squares method with kinematic constraints for the articulated object and a perspective camera model. We test our framework with an articulated robot and show results.

A Next Gen Interface for Embodied Learning

Emerging research from the learning sciences and human-computer interaction supports the premise that learning is effective when it is embodied, collaborative, and multimodal. In response, we have developed a mixed-reality environment called the Situated Multimedia Arts Learning Laboratory (SMALLab). SMALLab enables multiple students to interact with one another and digitally mediated elements via 3D movements and gestures in real physical space. It uses 3D object tracking, real time graphics, and surround-sound to enhance learning. We present two studies from the earth science domain that address questions regarding the feasibility and efficacy of SMALLab in a classroom context. We present data demonstrating that students learn more during a recent SMALLab intervention compared to regular classroom instruction. We contend that well-designed, mixed-reality environments have much to offer STEM learners, and that the learning gains transcend those that can be expected from more traditional classroom procedures.

SMALLab: virtual geology studies using embodied learning with motion, sound, and graphics

We present a new and innovative interface that allows the learner’s body to move freely in a multimodal learning environment. The Situated Multimedia Arts Learning Laboratory (SMALLab) uses 3D object tracking, real time graphics, and surround‐sound to enhance embodied learning. Our hypothesis is that optimal learning and retention occur when learning is embodied, multiple modalities are recruited, and learners are allowed to be highly collaborative. We present two geology studies in a high school setting. In the layer cake scenario, students learned more after three days of building and discussing depositional layers in SMALLab than would be expected from a simple retest effect. In the contour mapping scenario, students learned significantly more after creating and discussing maps in the SMALLab environment for three days, compared with a group that received three days of regular instruction with hands‐on projects. We contend that embodied, collaborative, virtual environments have much to offer STEM learn...

Correlation-based particle filter for 3D object tracking

This manuscript deals with the problem of 3D object tracking in a multisensor framework. The object is here described by a CAD model. It avoids any image preprocessing that leads, generally, to loss of information. We develop a particle filtering met

Simultaneously Execution of Target Detection, Posture Estimation, and Tracking of 3D Objects by Parametric Eigenspace Method Embedded Particle Filter

In this paper, an appearance based image processing method is proposed for target detection, posture estimation, and tracking of 3D objects by Parametric Eigenspace Method embedded particle filter. In this method, the computational cost of Parametric Eigenspace Method and the tracking error can be greatly reduced by using the outputs of previous time and the preliminary knowledge of dynamics of movement of targets for a particle filter. Namely, since the particles of the particle filter can be generated in the direction where the object will be moved by using the posture estimated by Parametric Eigenspace Method for the prediction model, the accuracy of the state estimation of the object can be improved without increasing the number of particles. Therefore, the proposed method uses output result of Particle Filters and Parametric Eigenspace Method recursively and mutually for fast simultaneous execution of detection, posture estimation, and tracking of the targets. Furthermore, we demonstrate validity of our approach by several experiments.

Three-dimensional tracking of occluded objects using integral imaging

We present experiments to illustrate tracking of occluded objects using three-dimensional (3D) integral imaging (II). Tracking of heavily occluded objects by conventional two-dimensional (2D) image processing may be difficult owing to the superposition of occlusion noise and object details. The effects of occlusion are remedied by 3D computational II reconstruction. We use a summation of absolute difference (SAD) algorithm between pixels of consecutive frames of a moving object for 3D tracking. SAD algorithm is not robust for 2D images of occluded objects; 3D computation reconstruction of the scene allows implementation of a SAD algorithm by reducing occlusion effects. Experimental results demonstrate 3D tracking of occluded objects. To the best of our knowledge, this is the first report on 3D tracking of objects using II.

Robust Real-Time 3D Object Tracking with Interfering Background Visual Projections

This paper presents a robust real-time object tracking system for human computer interaction in mediated environments with interfering visual projection in the background. Two major contributions are made in our research to achieve robust object tracking. A reliable outlier rejection algorithm is developed using the epipolar and homography constraints to remove false candidates caused by interfering background projections and mismatches between cameras. To reliably integrate multiple estimates of the 3D object positions, an efficient fusion algorithm based on mean shift is used. This fusion algorithm can also reduce tracking errors caused by partial occlusion of the object in some of the camera views. Experimental results obtained in real life scenarios demonstrate that the proposed system is able to achieve decent 3D object tracking performance in the presence of interfering background visual projection.

Implementation of disparity information‐based 3D object tracking

In this paper, a new 3D object tracking system using the disparity motion vector (DMV) is presented. In the proposed method, the time‐sequential disparity maps are extracted from the sequence of the stereo input image pairs and these disparity maps are used to sequentially estimate the DMV defined as a disparity difference between two consecutive disparity maps Similarly to motion vectors in the conventional video signals, the DMV provides us with motion information of a moving target by showing a relatively large change in the disparity values in the target areas. Accordingly, this DMV helps detect the target area and its location coordinates. Based on these location data of a moving target, the pan/tilt embedded in the stereo camera system can be controlled and consequently achieve real‐time stereo tracking of a moving target. From the results of experiments with 9 frames of the stereo image pairs having 256×256 pixels, it is shown that the proposed DMV‐based stereo object tracking system can track the moving target with a relatively low error ratio of about 3.05 % on average.

A new technique for real-time visual tracking of 3D objects

A new technique for real-time estimation of the pose of a moving object using a video camera is presented. A Binary Space Partition (BSP) tree is adopted to represent the object geometry and efficient selection techniques are developed to find an optimal set of feature points of the object to be used for feature extraction. The extended Kalman filter is used to estimate the pose of the object directly from the image measurements. The performance of the proposed approach is experimentally tested in the case of an object moving in the visible space of one fixed camera.

Shape statistics in kernel space for variational image segmentation

We present a variational integration of nonlinear shape statistics into a Mumford–Shah based segmentation process. The nonlinear statistics are derived from a set of training silhouettes by a novel method of density estimation which can be considered as an extension of kernel PCA to a probabilistic framework. We assume that the training data forms a Gaussian distribution after a nonlinear mapping to a higher-dimensional feature space. Due to the strong nonlinearity, the corresponding density estimate in the original space is highly non-Gaussian. Applications of the nonlinear shape statistics in segmentation and tracking of 2D and 3D objects demonstrate that the segmentation process can incorporate knowledge on a large variety of complex real-world shapes. It makes the segmentation process robust against misleading information due to noise, clutter and occlusion.

A 2D/3D model-based object tracking framework

This paper presents a robust framework for tracking complex objects in video sequences. Multiple hypothesis tracking (MHT) algorithm reported in (IEEE Trans. Pattern Anal. Mach. Intell. 18(2) (1996)) is modified to accommodate a high level representations (2D edge map, 3D models) of objects for tracking. The framework exploits the advantages of MHT algorithm which is capable of resolving data association/uncertainty and integrates it with object matching techniques to provide a robust behavior while tracking complex objects. To track objects in 2D, a 4D feature is used to represent edge/line segments and are tracked using MHT. In many practical applications 3D models provide more information about the object's pose (i.e., rotation information in the transformation space) which cannot be recovered using 2D edge information. Hence, a 3D model-based object tracking algorithm is also presented. A probabilistic Hausdorff image matching algorithm is incorporated into the framework in order to determine the geometric transformation that best maps the model features onto their corresponding ones in the image plane. 3D model of the object is used to constrain the tracker to operate in a consistent manner. Experimental results on real and synthetic image sequences are presented to demonstrate the efficacy of the proposed framework.

Real time tracking of 3D objects: an efficient and robust approach

The main aim of this article is the description of an original approach which allows the efficient tracking of two-dimensional (2D) patterns in image sequences. This approach consists of two stages. An off-line is devoted to the computation of an interaction matrix linking the aspect variation of a pattern with its displacement in the image. In a second stage, this matrix is used to track the pattern by the following process. At first, the position, the scale and the orientation of the pattern is predicted. Then, the difference between the pattern observed at the predicted position and the reference pattern (which is to be tracked) is computed. Finally, this difference multiplied by the interaction matrix gives a correction to apply to the prediction. Computing this correction has a very low computational cost, allowing a real time implementation of the algorithm. From these principles, we have developed a real time three-dimensional object tracker; objects are modeled by sets of 2D appearances. We show that these algorithms are resistant to occlusions and changes of illumination.

Visual tracking of a moving target by a camera mounted on a robot: a combination of control and vision

The authors present algorithms for robotic (eye-in-hand configuration) real-time visual tracking of arbitrary 3D objects traveling at unknown velocities in a 2D space (depth is given as known). Visual tracking is formulated as a problem of combining control with computer vision. A mathematical formulation of the control problem that includes information from a novel feedback vision sensor and represents everything with respect to the camera frame is presented. The sum-of-squared differences (SSD) optical flow is used to compute the vector of discrete displacements each instant of time. These displacements can be fed either directly to a PI (proportional-integral) controller or to a pole assignment controller or discrete steady-state Kalman filter. In the latter case, the Kalman filter calculates the estimated values of the system's states and the exogenous disturbances, and a discrete LQG (linear-quadratic Gaussian) controller computes the desired motion of the robotic system. The outputs of the controllers are sent to the Cartesian robotic controller. Performance results are presented.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>