Objects In Video Sequences Research Articles

Geodesy on label images, and applications to video sequence processing

Morphological operators based on the numerical ordering of grey-levels are used to filter regional minima and maxima in images. It is argued that label images constitute an appropriate framework for morphological filtering of regions which are neither minima nor maxima. Following a previous paper where we introduced morphological operations on label images [C. Ronse, V. Agnus, Morphology on label images: flat-type operators and connections, Journal of Mathematical Imaging and Vision 22 (2/3) (2005) 283–307], we study geodesic dilation (or erosion) and reconstruction. Since the lattice of label images is not distributive, some strange results may happen, so the standard definition of geodesic dilation and reconstruction must be modified in order to be effective; standard properties of geodesic operations are preserved only if we make some restrictions on the labels present in the mask or marker image. We give the relation between geodesic reconstruction and the flat zone connection on label images. We illustrate the theory with an application of morphology and geodesy on label images, to the segmentation of moving objects in video sequences.

Pearson-based mixture model for color object tracking

To track objects in video sequences, many studies have been done to characterize the target with respect to its color distribution. Most often, the Gaussian mixture model (GMM) is used to represent the object color density. In this paper, we propose to extend the normality assumption to more general families of distributions issued from the Pearson’s system. Precisely, we propose a method called Pearson mixture model (PMM), used in conjunction with Gaussian copula, which is dynamically updated to adapt itself to the appearance change of the object during the sequence. This model is combined with Kalman filtering to predict the position of the object in the next frame. Experimental results on gray-level and color video sequences show tracking improvements compared to classical GMM. Especially, the PMM seems robust to illumination variations, pose and scale changes, and also to partial occlusions, but its computing time is higher than the computing time of GMM.

RK-Means Clustering: K-Means with Reliability

This paper presents an RK-means clustering algorithm which is developed for reliable data grouping by introducing a new reliability evaluation to the K-means clustering algorithm. The conventional K-means clustering algorithm has two shortfalls: 1) the clustering result will become unreliable if the assumed number of the clusters is incorrect; 2) during the update of a cluster center, all the data points belong to that cluster are used equally without considering how distant they are to the cluster center. In this paper, we introduce a new reliability evaluation to K-means clustering algorithm by considering the triangular relationship among each data point and its two nearest cluster centers. We applied the proposed algorithm to track objects in video sequence and confirmed its effectiveness and advantages.

A new video segmentation method of moving objects based on blob-level knowledge

Variants of the background subtraction method are broadly used for the detection of moving objects in video sequences in different applications. In this work we propose a new approach to the background subtraction method which operates in the colour space and manages the colour information in the segmentation process to detect and eliminate noise. This new method is combined with blob-level knowledge associated with different types of blobs that may appear in the foreground. The idea is to process each pixel differently according to the category to which it belongs: real moving objects, shadows, ghosts, reflections, fluctuation or background noise. Thus, the foreground resulting from processing each image frame is refined selectively, applying at each instant the appropriate operator according to the type of noise blob we wish to eliminate. The approach proposed is adaptive, because it allows both the background model and threshold model to be updated. On the one hand, the results obtained confirm the robustness of the method proposed in a wide range of different sequences and, on the other hand, these results underline the importance of handling three colour components in the segmentation process rather than just the one grey-level component.

Segmenting, Modeling, and Matching Video Clips Containing Multiple Moving Objects

This paper presents a novel representation for dynamic scenes composed of multiple rigid objects that may undergo different motions and are observed by a moving camera. Multiview constraints associated with groups of affine-covariant scene patches and a normalized description of their appearance are used to segment a scene into its rigid components, construct three-dimensional models of these components, and match instances of models recovered from different image sequences. The proposed approach has been applied to the detection and matching of moving objects in video sequences and to shot matching, i.e., the identification of shots that depict the same scene in a video clip.

An Automated Video Object Extraction System Based on Spatiotemporal Independent Component Analysis and Multiscale Segmentation

Video content analysis is essential for efficient and intelligent utilizations of vast multimedia databases over the Internet. In video sequences, object-based extraction techniques are important for content-based video processing in many applications. In this paper, a novel technique is developed to extract objects from video sequences based on spatiotemporal independent component analysis (stICA) and multiscale analysis. The stICA is used to extract the preliminary source images containing moving objects in video sequences. The source image data obtained after stICA analysis are further processed using wavelet-based multiscale image segmentation and region detection techniques to improve the accuracy of the extracted object. An automated video object extraction system is developed based on these new techniques. Preliminary results demonstrate great potential for the new stICA and multiscale-segmentation-based object extraction system in content-based video processing applications.

Agent-Based Gesture Tracking

We describe an agent-based approach to the visual tracking of human hands and head that represents a very useful middle ground between the simple model-free tracking and the highly constrained model-based solutions. It combines the simplicity, speed, and flexibility of tracking without using explicit shape models with the ability to utilize domain knowledge and to apply various constraints characteristic of more elaborate model-based tracking approaches. One of the key contributions of our system, called AgenTrac, is that it unifies the power of data-fusion (cue integration) methodologies with a well-organized extended path-coherence-resolution approach designed to handle crossing trajectories of multiple objects. Both approaches are combined in an easily configurable framework. We are not aware of any path-coherence or data-fusion solution in the computer vision literature that equals the breadth, generality, and flexibility of our approach. The AgenTrac system is not limited to tracking only human motion; in fact, one of its main strengths is that it can be easily reconfigured to track many types of objects in video sequences. The multiagent paradigm simplifies the application of basic domain-specific constraints and makes the entire system flexible. The knowledge necessary for effective tracking can be easily encoded in agent hierarchies and agent interactions.

Morphology on Label Images: Flat-Type Operators and Connections

We consider morphological operations on images whose pixel values are considered as labels without ordering between them, except for a least element ? (meaning no label) and a greatest element ?(Meaning conflicting labels). Flat dilations and erosions can be defined as in usual grey-level morphology. Since the lattice of label images is not distributive, non-flat operators can be obtained by combination of flat ones. Given any connectivity on sets, there is a connection on label images for which the connected components of an image correspond precisely to its flat zones with their labels attached. Some specific applications of label morphology are given. In the sequel of this paper [20], we will examine geodesic dilations and reconstructions on label images, and show how this variant of mathematical morphology can be applied to the segmentation of moving objects in video sequences [2, 3].

A snake model for object tracking in natural sequences

Tracking moving objects in video sequences is a task that emerges in various fields of study: video analysis, computer vision, biomedical systems, etc. In the last decade, special attention has been drawn to problems concerning tracking in real-world environments, where moving objects do not obey any afore-known constraints about their nature and motion or the scenes they are moving in. Apart from the existence of noise and environmental changes, many problems are also concerned, due to background texture, complicated object motion, and deformable and/or articulated objects, changing their shape while moving along time. Another phenomenon in natural sequences is the appearance of occlusions between different objects, whose handling requires motion information and, in some cases, additional constraints. In this work, we revisit one of the most known active contours, the Snakes, and we propose a motion-based utilization of it, aiming at successful handling of the previously mentioned problems. The use of the object motion history and first order statistical measurements of it, provide us with information for the extraction of uncertainty regions, a kind of shape prior knowledge w.r.t. the allowed object deformations. This constraining also makes the proposed method efficient, handling the trade-off between accuracy and computation complexity. The energy minimization is approximated by a force-based approach inside the extracted uncertainty regions, and the weights of the total snake energy function are automatically estimated as respective weights in the resulting evolution force. Finally, in order to handle background complexity and partial occlusion cases, we introduce two rules, according to which the moving object region is correctly separated from the background, whereas the occluded boundaries are estimated according to the object's expected shape. To verify the performance of the proposed method, some experimental results are included, concerning different cases of object tracking, indoors and outdoors, with rigid and deformable objects, noisy and textured backgrounds, as well as appearance of occlusions.

A 2D/3D model-based object tracking framework

This paper presents a robust framework for tracking complex objects in video sequences. Multiple hypothesis tracking (MHT) algorithm reported in (IEEE Trans. Pattern Anal. Mach. Intell. 18(2) (1996)) is modified to accommodate a high level representations (2D edge map, 3D models) of objects for tracking. The framework exploits the advantages of MHT algorithm which is capable of resolving data association/uncertainty and integrates it with object matching techniques to provide a robust behavior while tracking complex objects. To track objects in 2D, a 4D feature is used to represent edge/line segments and are tracked using MHT. In many practical applications 3D models provide more information about the object's pose (i.e., rotation information in the transformation space) which cannot be recovered using 2D edge information. Hence, a 3D model-based object tracking algorithm is also presented. A probabilistic Hausdorff image matching algorithm is incorporated into the framework in order to determine the geometric transformation that best maps the model features onto their corresponding ones in the image plane. 3D model of the object is used to constrain the tracker to operate in a consistent manner. Experimental results on real and synthetic image sequences are presented to demonstrate the efficacy of the proposed framework.

ACTIVE CONTOUR MODEL USING WAVELET MODULUS FOR OBJECT SEGMENTATION AND TRACKING IN VIDEO SEQUENCES

We propose an integrated wavelet-based framework of the active contour model (snake) for segmentation and motion tracking of deformable objects in video sequences. The input image frame is represented by means of wavelet transform. First, the moduli of wavelet transform coefficients are used in a multi-resolution motion estimation process to find the initial snake contour in the current frame. The presented multi-resolution motion estimation methods allow a larger movement of the tracked object than does traditional image-based motion estimation. Secondly, the wavelet transform modulus at each scale is considered in the energy function of the snake model. The snake computation is based on a coarse-to-fine scale continuation method. Application of the proposed methods to biological cell tracking is demonstrated in experiments.

Automatic segmentation of moving objects in video sequences: a region labeling approach

The emerging video coding standard MPEG-4 enables various content-based functionalities for multimedia applications. To support such functionalities, as well as to improve coding efficiency, MPEG-4 relies on a decomposition of each frame of an image sequence into video object planes (VOP). Each VOP corresponds to a single moving object in the scene. This paper presents a new method for automatic segmentation of moving objects in image sequences for VOP extraction. We formulate the problem as graph labeling over a region adjacency graph (RAG), based on motion information. The label field is modeled as a Markov random field (MRF). An initial spatial partition of each frame is obtained by a fast, floating-point based implementation of the watershed algorithm. The motion of each region is estimated by hierarchical region matching. To avoid inaccuracies in occlusion areas, a novel motion validation scheme is presented. A dynamic memory, based on object tracking, is incorporated into the segmentation process to maintain temporal coherence of the segmentation. Finally, a labeling is obtained by maximization of the a posteriori probability of the MRF using motion information, spatial information and the memory. The optimization is carried out by highest confidence first (HCF). Experimental results for several video sequences demonstrate the effectiveness of the proposed approach.

A 3-Step Algorithm Using Region-Based Active Contours for Video Objects Detection

We propose a 3-step algorithm for the automatic detection of moving objects in video sequences using region-based active contours. First, we introduce a very full general framework for region-based active contours with a new Eulerian method to compute the evolution equation of the active contour from a criterion including both region-based and boundary-based terms. This framework can be easily adapted to various applications, thanks to the introduction of functions named descriptors of the different regions. With this new Eulerian method based on shape optimization principles, we can easily take into account the case of descriptors depending upon features globally attached to the regions. Second, we propose a 3-step algorithm for detection of moving objects, with a static or a mobile camera, using region-based active contours. The basic idea is to hierarchically associate temporal and spatial information. The active contour evolves with successively three sets of descriptors: a temporal one, and then two spatial ones. The third spatial descriptor takes advantage of the segmentation of the image in intensity homogeneous regions. User interaction is reduced to the choice of a few parameters at the beginning of the process. Some experimental results are supplied.

Stand-Alone Objective Segmentation Quality Evaluation

The identification of objects in video sequences, that is, video segmentation, plays a major role in emerging interactive multimedia services, such as those enabled by the ISO MPEG-4 and MPEG-7 standards. In this context, assessing the adequacy of the identified objects to the application targets, that is, evaluating the segmentation quality, assumes a crucial importance. Video segmentation technology has received considerable attention in the literature, with algorithms being proposed to address various types of applications. However, the segmentation quality performance evaluation of those algorithms is often ad hoc, and a well-established solution is not available. In fact, the field of objective segmentation quality evaluation is still maturing; recently, some more efforts have been made, mainly following the emergence of the MPEG object-based coding and description standards. This paper discusses the problem of objective segmentation quality evaluation in its most difficult scenario: stand-alone evaluation, that is when a reference segmentation is not available for comparative evaluation. In particular, objective metrics are proposed for the evaluation of stand-alone segmentation quality for both individual objects and overall segmentation partitions.

Fast and automatic video object segmentation and tracking for content-based applications

The new video-coding standard MPEG-4 enables content-based functionality, as well as high coding efficiency, by taking into account shape information of moving objects. A novel algorithm for segmentation of moving objects in video sequences and extraction of video object planes (VOPs) is proposed . For the case of multiple video objects in a scene, the extraction of a specific single video object (VO) based on connected components analysis and smoothness of VO displacement in successive frames is also discussed. Our algorithm begins with a robust double-edge map derived from the difference between two successive frames. After removing edge points which belong to the previous frame, the remaining edge map, moving edge (ME), is used to extract the VOP. The proposed algorithm is evaluated on an indoor sequence captured by a low-end camera as well as MPEG-4 test sequences and produces promising results.

サッカー番組における動作カテゴリー化処理を伴う動画検索手法

Focusing on soccer programs, we have developed a method of retrieving scenes by categorizing the motion of important objects like a goalkeeper. It is designed for an object-based video database system that manages information about video objects in video sequences. The scene retrieval consists of two steps for the comparison process : (I) categorizing the motion and (II) calculating the similarity in distance to a reference video example of the same category. Motion categorization is carried out by probabilistically learning the sequences of motion parameters for a goalkeeper. We have implemented this method in an object relational database system containing various soccer scenes and verified it.

Spatio-temporal composition of video objects: representation and querying in video database systems

A key characteristic of video data is the associated spatial and temporal semantics. It is important that a video model models the characteristics of objects and their relationships in time and space. J.F. Allen's (1983) 13 temporal relationships are often used in formulating queries that contain the temporal relationships among video frames. For the spatial relationships, most of the approaches are based on projecting objects on a two or three-dimensional coordinate system. However, very few attempts have been made formally to represent the spatio-temporal relationships of objects contained in the video data and to formulate queries with spatio-temporal constraints. The purpose of the work is to design a model representation for the specification of the spatio-temporal relationships among objects in video sequences. The model describes the spatial relationships among objects for each frame in a given video scene and the temporal relationships (for this frame) of the temporal intervals measuring the duration of these spatial relationships. It also models the temporal composition of an object, which reflects the evolution of object's spatial relationships over the subsequent frames in the video scene and in the entire video sequence. Our model representation also provides an effective and expressive way for the complete and precise specification of distances among objects in digital video. This model is a basis for the annotation of raw video.

Multi-Object Tracking in Video

This paper reports on tracking of multiple objects using color histogram backprojection and motion cues. Four tasks which facilitate this are discussed. The first is an adaptive color histogram backprojection (which builds upon the works of Swain and Ballard) and its application to tracking of multiple objects in video sequences. The second task is designing efficient fast blob detectors for selecting regions of interest in video sequences. The third is motion detection based on color histogram backprojection. Achieving these tasks led to multi-objects tracking. Various video sequences were used to demonstrate effective tracking of multiple objects. Notably, we created an interactive multiple objects tracker (CLICK-IT) which in its present form is set at three objects but can be extended easily. CLICK-IT (CSIRO Laboratory for Imaging by Content and Knowledge—Interactive Television) is a PC-based system which provides the user with an intelligent highlighter pen for sports action replay. It is intended as a truly interactive improvement on the drawing pad technology currently used for video annotation in sports broadcasting. The system uses computer vision techniques to focus attention and track particular objects (player(s), ball, horse(s), …) and semi-automatically annotate the dynamic scene. This paper describes the system including the user interface, the tracking technology based on color and motion information, and system performance evaluation in applications to surveillance-like sequences, running, rugby league football, basketball and soccer. Finally, video scene detection based on color histogram is discussed.

A noise robust method for 2D shape estimation of moving objects in video sequences considering a moving camera

For the coding of image sequences at very low bit rates, it has been shown that the coding efficiency can be increased by taking into account the shape of moving objects in the scene. Moreover, the upcoming ISO/MPEG-4 standard considers the 2D shape of moving objects not only for reasons of coding efficiency, but also to provide the user with so-called content-based functionalities. However, the perfect automatic segmentation of moving objects in image sequences is still an unsolved problem. In this paper, an algorithm for automatic, noise robust 2D shape estimation of moving objects in video sequences is presented, which considers a moving camera. The algorithm consists of four main steps. In the first step, a possibly apparent camera motion is estimated and compensated. By the second step, a possibly apparent scene cut is detected, and if necessary the segmentation algorithm is reset. In the third step, a change detection mask is estimated by a relaxation technique, using local thresholds which consider the state of neighbouring pels. By the fourth step, regions where the background has been uncovered by a moving object are estimated, using motion information from a displacement vector field. Finally, the resulting object mask is adapted to luminance edges in the corresponding image, in order to improve the shape accuracy. Furthermore, the temporal coherency of the object mask is improved by applying a memory. The proposed algorithm is compared to an approach for 2D shape estimation of moving objects from the literature, which does not use any edge adaptation or object memory and which uses only one global threshold. The experimental results show that the resulting object shapes from the proposed algorithm look subjectively much better than those from the reference algorithm.

Tracking multiple nonrigid objects in video sequences

This paper presents a method to track multiple nonrigid objects in video sequences. First, we present related works on tracking methods. Second, we describe our proposed approach. We use the notion of target to represent the perception of object motion. To handle the particularities of nonrigid objects we define a target as an individually tracked moving region or as a group of moving regions globally tracked. Then we explain how to compute the trajectory of a target and how to compute the correspondences between known targets and moving regions newly detected. In the case of an ambiguous correspondence we define a compound target to freeze the associations between targets and moving regions until a more accurate information is available. Finally we provide an example to illustrate the way we have implemented the proposed tracking method for video-surveillance applications.