Motion Of Nonrigid Objects Research Articles

Multi-camera multi-object tracking: A review of current trends and future advances

The nascent applicability of multi-camera tracking (MCT) in numerous real-world applications makes it a significant computer vision problem. While visual tracking of objects, especially in video obtained from single camera setup, has drawn huge research attention, the constant identification and tracking of targets as they transit across multiple cameras remains an open research problem. In addition to the linking of target appearance and trajectory information across frames, effective association of such data across multiple cameras is also very critical in MCT. Occlusion, appearance variability, camera motion, as well as nonrigid object structure and motion are widely recognized constraints and major sources of concerns in MCT. In recent years, several literatures have been contributed suggesting a variety of approaches to addressing various problems in MCT. However, studies that critically review and report the advances and trends of research in MCT are still limited. This current study presents a comprehensive and up-to-date review of visual object tracking in multi-camera settings. In this paper, we analyze and categorize existing works based on six crucial facets: problem formulation, adopted problem solving approach, data association requirements, mutual exclusion constraints, benchmark datasets, and performance metrics. Furthermore, the study summarizes the outcomes of 30 state-of-the-art MCT algorithms on common datasets to allow quantitative comparison and analysis of their experimental results. Finally, we examine recent advances in MCT and suggest some promising future research directions.

Graph Matching for Marker Labeling and Missing Marker Reconstruction With Bone Constraint by LSTM in Optical Motion Capture

Optical motion capture (MOCAP) is a commonly used technology to record the motion of non-rigid objects with high accuracy in 3D space. However, the MOCAP data has to be processed further before it can be used. The scattered reconstructed motion data must constitute a human configuration by labelling process according to the predefined template, and the missing markers have to be reconstructed to produce a stable motion trajectory. In this work, we propose a novel labelling method for motion sequences. First, a novel graph matching method is employed to determine the connection relationship of the scattered motion data for a single frame. Then, Kalman filtering is used for tracking in the motion sequence. As for the challenge coming from missing markers, we propose a new motion data preprocessing method considering the bone length constraint, which represents the information of variation in the relative position of adjacent markers. The processed motion data is input into a Long-Short Term Memory (LSTM) model to recover the missing markers and de-noise the motion data. The experiment conducted on our own dataset proves that our labelling method achieves a similar effect to Cortex, which is a commonly used commercial motion data analysis software. The experiment on CMU dataset demonstrates that our missing marker reconstruction method can achieve an art-of-state result. The labelling code will be pulished on https://github.com/Lijianfang6930/Graph-Matching-for-Marker-Labelling

Cross-modality motion parameterization for fine-grained video prediction

While predicting video content is challenging given the huge unconstrained searching space, this work explores cross-modality constraints to safeguard the video generation process and seeks improved content prediction. By observing the underlying correspondence between the sound and the object movement, we propose a novel cross-modality video generation network. Via adversarial training, this network directly links sound with the movement parameters of the operated object and automatically outputs corresponding object motion according to the rhythm of the given audio signal. We experiment on both rigid object and non-rigid object motion prediction tasks and show that our method significantly reduces motion uncertainty for the generated video content, with the guidance of the associated audio information.

Video Object Segmentation and Tracking Framework With Improved Threshold Decision and Diffusion Distance

Video object segmentation and tracking are two essential building blocks of smart surveillance systems. However, there are several issues that need to be resolved. Threshold decision is a difficult problem for video object segmentation with a multibackground model. In addition, some conditions make robust video object tracking difficult. These conditions include nonrigid object motion, target appearance variations due to changes in illumination, and background clutter. In this paper, a video object segmentation and tracking framework is proposed for smart cameras in visual surveillance networks with two major contributions. First, we propose a robust threshold decision algorithm for video object segmentation with a multibackground model. Second, we propose a video object tracking framework based on a particle filter with the likelihood function composed of diffusion distance for measuring color histogram similarity and motion clue from video object segmentation. The proposed framework can track nonrigid moving objects under drastic changes in illumination and background clutter. Experimental results show that the presented algorithms perform well for several challenging sequences, and our proposed methods are effective for the aforementioned issues.

Learned Non-Rigid Object Motion is a View-Invariant Cue to Recognizing Novel Objects

There is evidence that observers use learned object motion to recognize objects. For instance, studies have shown that reversing the learned direction in which a rigid object rotated in depth impaired recognition accuracy. This motion reversal can be achieved by playing animation sequences of moving objects in reverse frame order. In the current study, we used this sequence-reversal manipulation to investigate whether observers encode the motion of dynamic objects in visual memory, and whether such dynamic representations are encoded in a way that is dependent on the viewing conditions. Participants first learned dynamic novel objects, presented as animation sequences. Following learning, they were then tested on their ability to recognize these learned objects when their animation sequence was shown in the same sequence order as during learning or in the reverse sequence order. In Experiment 1, we found that non-rigid motion contributed to recognition performance; that is, sequence-reversal decreased sensitivity across different tasks. In subsequent experiments, we tested the recognition of non-rigidly deforming (Experiment 2) and rigidly rotating (Experiment 3) objects across novel viewpoints. Recognition performance was affected by viewpoint changes for both experiments. Learned non-rigid motion continued to contribute to recognition performance and this benefit was the same across all viewpoint changes. By comparison, learned rigid motion did not contribute to recognition performance. These results suggest that non-rigid motion provides a source of information for recognizing dynamic objects, which is not affected by changes to viewpoint.

Object Modelling and Tracking in Videos via Multidimensional Features

We propose to model a tracked object in a video sequence by locating a list of object features that are ranked according to their ability to differentiate against the image background. The Bayesian inference is utilised to derive the probabilistic location of the object in the current frame, with the prior being approximated from the previous frame and the posterior achieved via the current pixel distribution of the object. Consideration has also been made to a number of relevant aspects of object tracking including multidimensional features and the mixture of colours, textures, and object motion. The experiment of the proposed method on the video sequences has been conducted and has shown its effectiveness in capturing the target in a moving background and with nonrigid object motion.

Multiple View Geometry for Curvilinear Motion Cameras

This paper introduces a tensorial representation of multiple cameras with arbitrary curvilinear motions. It enables us to define a multilinear relationship among image points derived from non-rigid object motions viewed from multiple cameras with arbitrary curvilinear motions. We show the new multilinear relationship is useful for generating images and reconstructing 3D non-rigid object motions viewed from cameras with arbitrary curvilinear motions. The method is tested in real image sequences.

Multiple View Geometry under Projective Projection in Space-Time

This paper introduces multiple view geometry under projective projection from four-dimensional, space to two-dimensional space which can represent multiple view geometry under the projection of space-time. We show the multifocal tensors defined under space-time projective projection can be derived from non-rigid object motions viewed from multiple cameras with arbitrary translational motions, and they are practical for generating images of non-rigid object motions viewed from cameras with arbitrary translational motions. The method is tested in real image sequences.

Motion estimation for cardiac emission tomography by optical flow methods

This paper describes a new method for estimating the 3D, non-rigid object motion in a time sequence of images. The method is a generalization of a standard optical flow algorithm that is incorporated into a successive quadratic approximation framework. The method was evaluated for gated cardiac emission tomography using images obtained from a mathematical, 4D phantom and a physical, dynamic phantom. The results showed that the proposed method offers improved motion estimation accuracy relative to the standard optical flow method. Convergence of the proposed algorithm was evidenced with a monotonically decreasing objective function value with iteration. Practical application of the motion estimation method in cardiac emission tomography includes quantitative myocardial motion estimation and 4D, motion-compensated image reconstruction.

Rotation constrained power factorization for structure from motion of nonrigid objects

The paper addresses the problem of recovering 3D structure and motion of nonrigid objects from image sequences. We propose a rotation constrained power factorization (RCPF) algorithm that combines the orthonormality and the replicated block structure of the motion matrix directly into iterations. The algorithm overcomes some limitations of previous SVD-based methods and can work with missing data. Based on the shape bases of the batch-type factorization, we also propose a sequential factorization technique to recover the shape and motion of new frames conveniently. Extensive experiments show the effectiveness of the proposed algorithm.

Structure and motion of nonrigid object under perspective projection

The paper focuses on the problem of structure and motion of nonrigid object from image sequence under perspective projection. Many previous methods on this problem utilize the extension technique of SVD factorization based on rank constraint to the tracking matrix, where the 3D shape of nonrigid object is expressed as a weighted combination of a set of shape bases. All these solutions are based on the assumption of Affine camera model. This assumption will become invalid and cause large reconstruction errors when the object is close to the camera. In this paper, we propose two algorithms, namely the linear recursive estimation and the nonlinear optimization, to extend these methods to general perspective camera model. Both algorithms are based on the shape and motion of weak perspective projection. The former one updates the solutions from weak perspective to perspective projection by refining the scalars corresponding to the projective depths recursively. The latter one is based on nonlinear optimization by minimizing the perspective reprojection residuals. Extensive experiments on simulated data and real image sequences are performed to validate the effectiveness of our new algorithms and noticeable improvements over the previous solutions are observed.

Gaussian and Laplacian of Gaussian weighting functions for robust feature based tracking

Object tracking algorithms found extensively in the computer vision literature either are inhibited by various assumptions such as simplicity of motion and shape characteristics of objects or are overly sensitive to noise. We propose and successfully test two new weighting functions for a feature-based object-tracking algorithm to achieve superior performance in tracking motion of non-rigid objects under noisy conditions. We present the implications of using the weighting functions in real and synthetic image sequences to overcome the noise produced at acquisition source (charge coupled device—CCD), or in the background environment. We also present a mechanism for determining the optimal weighting function based on image parameters, more specifically the edge characteristics of objects in the image.

Spatio-temporal shape building from image sequences using lateral interaction in accumulative computation

To be able to understand the motion of non-rigid objects, techniques in image processing and computer vision are essential for motion analysis. Lateral interaction in accumulative computation for extracting non-rigid shapes from an image sequence has recently been presented, as well as its application to segmentation from motion. In this paper, we introduce a modified version of the first multi-layer architecture. This version uses the basic parameters of the LIAC model to spatio-temporally build up to the desired extent the shapes of all moving objects present in a sequence of images. The influences of LIAC model parameters are explained in this paper, and we finally show some examples of the usefulness of the model proposed.

Segmentation from motion of non-rigid objects by neuronal lateral interaction

The problem we are stating is the discrimination of non-rigid objects capable of holding our attention in a scene. Motion allows gradually obtaining all moving objects shapes. We introduce an algorithm that fuses spots obtained by means of neuronal lateral interaction in accumulative computation.

Tracking nonrigid motion and structure from 2D satellite cloud images without correspondences

Tracking both structure and motion of nonrigid objects from monocular images is an important problem in vision. In this paper, a hierarchical method which integrates local analysis (that recovers small details) and global analysis (that appropriately limits possible nonrigid behaviors) is developed to recover dense depth values and nonrigid motion from a sequence of 2D satellite cloud images without any prior knowledge of point correspondences. This problem is challenging not only due to the absence of correspondence information but also due to the lack of depth cues in the 2D cloud images (scaled orthographic projection). In our method, the cloud images are segmented into several small regions and local analysis is performed for each region. A recursive algorithm is proposed to integrate local analysis with appropriate global fluid model constraints, based on which a structure and motion analysis system, SMAS, is developed. We believe that this is the first reported system in estimating dense structure and nonrigid motion under scaled orthographic views using fluid model constraints. Experiments on cloud image sequences captured by meteorological satellites (GOES-8 and GOES-9) have been performed using our system, along with their validation and analyses. Both structure and 3D motion correspondences are estimated to subpixel accuracy. Our results are very encouraging and have many potential applications in earth and space sciences, especially in cloud models for weather prediction.

Unsupervised segmentation of color-texture regions in images and video

A method for unsupervised segmentation of color-texture regions in images and video is presented. This method, which we refer to as JSEG, consists of two independent steps: color quantization and spatial segmentation. In the first step, colors in the image are quantized to several representative classes that can be used to differentiate regions in the image. The image pixels are then replaced by their corresponding color class labels, thus forming a class-map of the image. The focus of this work is on spatial segmentation, where a criterion for "good" segmentation using the class-map is proposed. Applying the criterion to local windows in the class-map results in the "J-image," in which high and low values correspond to possible boundaries and interiors of color-texture regions. A region growing method is then used to segment the image based on the multiscale J-images. A similar approach is applied to video sequences. An additional region tracking scheme is embedded into the region growing process to achieve consistent segmentation and tracking results, even for scenes with nonrigid object motion. Experiments show the robustness of the JSEG algorithm on real images and video.

Region-Based Video Coding Using a Geometric Motion Compensation

A new region-based video coding technique, which combines region segmentation and geometric motion estimation, is proposed in this paper. The region segmentation algorithm based on both the histogram concavities and the probabilistic relaxation is applied to extract the significant regions. The merging of regions in the segmentation depends on the attributes of the regions, besides the motion vectors. A new approach based on the geometric features of objects and the scalable translation invariant rotation-to-shifting (STIRS) signatures is applied to the motion estimation of the global regions. Vector quantization (VQ) techniques are employed to solve the problems due to nonrigid object motions. With these techniques, the critical regions can be extracted with good performance and relatively low complexity. Furthermore, an uncovered/overlapped motion compensation method is presented. The integral algorithm is compared to Yokoyama's algorithm (Yokoyama et al., 1995, IEEE Trans. Circuit Systems Video Technol.5, 500–507). Our simulations show improved performance, in terms of both SNR improvement and reduction of computational requirements. For example, we achieve about 15% reduction of computational time and about 0.8 dB improvement in SNR for the “Miss America sequence.” The frame-to-frame SNR variation is also much smaller, which implies that the visual quality between frames is also very stable.

Nonrigid Motion Analysis: Articulated and Elastic Motion

Motion of physical objects in the world is, in general, nonrigid. In robotics and computer vision, the motion of nonrigid objects is of growing interest to researchers from a wide spectrum of disciplines. The nonrigid objects being studied may be generally categorized into three groups according to the degree of deformation of body parts: articulated, elastic, and fluid. In articulated motion, individual rigid parts of an object move independently of one another and the motion of the whole object is nonrigid in nature. Elastic motion is nonrigid motion that conforms to a certain degree of continuity or smoothness. Fluid motion violates even the continuity assumption and may involve topological variations and turbulent deformations. This paper presents an overview of existing work on articulated and elastic motion, motivated by problems relating to the motion of the human body and of an animal heart, respectively. We study various approaches for recovering the 3D structure and motion of objects through a sequence of images in a bottom-up fashion, a strategy widely employed by various investigators. These approaches are classified as (1) motion recovery without shape models, and (2) model-based analysis. In the discussion of each algorithm, we also include a description of the complexity of feature and motion constraints, which are highly related to each other.

A vision-based framework for the discovery-driven manipulation of non-rigid objects

Since the analytical expressions for the representation of non-rigid object structure and motion are severely underconstrained, current techniques for non-rigid object manipulation employ physical object models known prior to sensing. Recently, however, psychophysical studies have revealed that humans are able to discover proper motor coordination skills through sensory input without the use of previously known physical models. In this paper, a robust, discovery-driven, vision-based framework for the robotic manipulation of non-rigid objects is developed and experimentally verified using various flexible linear objects. Employing the novel concept of relative elasticity, the algorithms derived using this framework are completely sensor-based, requiring the use of no a priori explicit, physics-based models.

Nonrigid motion tracking of image sequences based on smoothness constraints

AbstractTo estimate the motion and deformation of nonrigid objects accurately from image sequences, this paper proposes a method for nonrigid motion tracking based on smoothness constraints, which have an effect on noise reduction in optical flow computation by a gradient‐based method. The motion of nonrigid objects generally is assumed to be smooth according to such physical properties as inertia and elasticity. Therefore, such smoothness constraints are defined in space (in frame) and time (between frames) as an active net model in formulating an energy functional and Lagrange equation, and the optical flow field derived from image sequences by shape preservation and natural deformation is modified. In addition, applying continuous Kalman filtering theory for long sequences of image frames, it is possible to estimate the temporal continuous deformation of elastic objects. The nonrigid motion tracking is demonstrated by experiments involving the modification of the shape distortion yielded by noise and the reconstruction of the real motion and deformation.