Segmentation and tracking of video objects for a content-based video indexing context

This paper examines the problem of segmentation and tracking of video objects for a content-based information retrieval context. Our method starts first with an interactive video object selection, then alternately tracks and fits the object of interest as long as possible. A user-based selection is required in order to initialize the process, whereas an active contour model progressively refines the selection by fitting the natural edges of the object. The video object is thus tracked by using a hybrid structure combining a hierarchical mesh for the motion estimation between two frames and a multi-resolution active contour model. This contour model is derived directly from the mesh boundaries in order to reposition the snake's nodes onto the natural edges of the object.

Video object segmentation and tracking are two essential building blocks of smart surveillance systems. However, there are several issues that need to be resolved. Threshold decision is a difficult problem for video object segmentation with a multi-background model. In addition, some conditions make robust video object tracking difficult. These conditions include nonrigid object motion, target appearance variations due to changes in illumination, and background clutter. In this paper, a video object segmentation and tracking framework is proposed for smart cameras in visual surveillance networks with two major contributions. First, we propose a robust threshold decision algorithm for video object segmentation with a multi-background model. Second, we propose a video object tracking framework based on a particle filter with the likelihood function composed of diffusion distance for measuring color histogram similarity and motion clue from video object segmentation. The proposed framework can track nonrigid moving objects under drastic changes in illumination and background clutter. Experimental results show that the presented algorithms perform well for several challenging sequences, and our proposed methods are effective for the aforementioned issues.

Segmenting and tracking of objects in video is of great importance for video-based encoding, surveillance, and retrieval. However, the inherent difficulty of object segmentation and tracking is to distinguish changes in the displacement of objects from disturbing effects such as noise and illumination changes. Therefore, in this paper, we formulate a color-based deformable model which is robust against noisy data and changing illumination. Computational methods are presented to measure color constant gradients. Further, a model is given to estimate the amount of sensor noise through these color constant gradients. The obtained uncertainty is subsequently used as a weighting term in the deformation process. Experiments are conducted on image sequences recorded from three-dimensional scenes. From the experimental results, it is shown that the proposed color constant deformable method successfully finds object contours robust against illumination, and noisy, but homogeneous regions.

Object segmentation and object tracking are fundamental research areas in the computer vision community. These two topics are difficult to handle some common challenges, such as occlusion, deformation, motion blur, scale variation, and more. The former contains heterogeneous object, interacting object, edge ambiguity, and shape complexity; the latter suffers from difficulties in handling fast motion, out-of-view, and real-time processing. Combining the two problems of Video Object Segmentation and Tracking (VOST) can overcome their respective difficulties and improve their performance. VOST can be widely applied to many practical applications such as video summarization, high definition video compression, human computer interaction, and autonomous vehicles. This survey aims to provide a comprehensive review of the state-of-the-art VOST methods, classify these methods into different categories, and identify new trends. First, we broadly categorize VOST methods into Video Object Segmentation (VOS) and Segmentation-based Object Tracking (SOT). Each category is further classified into various types based on the segmentation and tracking mechanism. Moreover, we present some representative VOS and SOT methods of each time node. Second, we provide a detailed discussion and overview of the technical characteristics of the different methods. Third, we summarize the characteristics of the related video dataset and provide a variety of evaluation metrics. Finally, we point out a set of interesting future works and draw our own conclusions.

Video object tracking and segmentation with box annotation

This paper presented a video moving object segmentation and tracking system based on the active contour and the color classification models. First, the active contour model is applied to segment the target object in the initial frame. From the segmented object, the object and background regions are extracted. Then the object and the background regions are separately clustered according to color feature by using the K-means algorithm. Subsequently, the video object in the next frame is automatically tracked by using temporal differencing and block matching. The moving and stationary regions in a frame are estimated by the temporal differencing. In the moving regions, pixels are obtained their classification from the previous frame using block matching while they are directly received their classification from the previous frame in the stationary regions. Experimental results show that the proposed method provides better performance than the active contour method applied in video object tracking.

As a critical step in many multimedia applications, shot boundary detection has attracted many research interests in recent years. The most of existing methods measure the similarity among video frames based on its low-level feathers. However, they are sensitive to the change in not only brightness, color, motion of object, but also camera motions and the quality of video. This paper proposes an innovative shot boundary detection method for news video based on video object segmentation and tracking. It combines three main techniques: the partitioned histogram comparison method, the video object segmentation and tracking based on wavelet analysis. The partitioned histogram comparison is used as the first filter to effectively reduce the number of video frames which need object segmentation and tracking. The unsupervised video object segmentation and tracking based on wavelet analysis is robust to those problems mentioned above. The efficacy of the proposed method is extensively tested with more than 3 hours of CCTV and CNN news programs, and that 96.4% recall with 97.2% precision have been achieved.

Moving object segmentation and detection have become an important topic in computer perspective. As such, it is widely used in video surveillance such as driving assistance program, robots, traffic monitoring, and crime pattern identification. In addition, video object tracking is an important function in video surveillance systems because it provides temporary interactive information about moving objects. An important function of video object segmentation is to find and separate important elements in the video frame behind the domain. The purpose of video tracking is to combine targeted objects into consecutive video frames. First of all, enhanced threshold filtered video object detection and tracking (TFVODT) is designed to classify objects according to their size, color, and to get better accuracy of video object detection. Initially, the TFVODT framework distinguishes a video object by its characteristics such as size and color. The TFVODT framework performs the function of distinguishing an object through the median filter-based enhanced Laplacian thresholding process. Along with the support of the split object, the TFVODT framework does well to track the video object. Second, threshold filtered video object detection and tracking (ITFVODT) is developed to distinguish video’s elements based on their features such as texture, durability, and performance of video object detection. All video frames found in the ITFVODT framework contain the similar features as quality and contrast.KeywordsObject trackingITFVODTTFVODTEMFVDSegmentation

Multiple existing benchmarks involve tracking and segmenting objects in video e.g., Video Object Segmentation (VOS) and Multi-Object Tracking and Segmentation (MOTS), but there is little interaction between them due to the use of disparate benchmark datasets and metrics (e.g. $\mathcal{J}\& {\mathcal{F}}$, mAP, sMOTSA). As a result, published works usually target a particular benchmark, and are not easily comparable to each another. We believe that the development of generalized methods that can tackle multiple tasks requires greater cohesion among these research sub-communities. In this paper, we aim to facilitate this by proposing BURST, a dataset which contains thousands of diverse videos with high-quality object masks, and an associated benchmark with six tasks involving object tracking and segmentation in video. All tasks are evaluated using the same data and comparable metrics, which enables researchers to consider them in unison, and hence, more effectively pool knowledge from different methods across different tasks. Additionally, we demonstrate several baselines for all tasks and show that approaches for one task can be applied to another with a quantifiable and explainable performance difference. Dataset annotations are available at: https://github.com/Ali2500/BURST-benchmark.

Object segmentation, detection and tracking in videos is one of the most important task of computer vision. It is necessary in all of the real time deployed surveillance systems. Various unsupervised and semi-supervised video object segmentation techniques have been implemented and shown efficient results. But all of these techniques process all of the frames of a video sequence, which requires a huge training data and results in a large computational time. In this paper, a semi-supervised technique is proposed which segments an object in a video by just processing a single frame of the sequence. In this framework, a fully convolutional network is used to separate the foreground from the image, create the mask of the object and then segments the object with the help of this mask. The foreground separation in a frame is done by using pre-trained network while, training and testing of rest of the network is done using a specified dataset named as DAVIS. The results show that, the proposed framework takes less computational time and has also improved the overall accuracy of video object segmentation by 10% as compared to previous techniques.

Video object segmentation and tracking are essential for content-based video processing. This paper presents a framework for a semiautomatic approach to this problem. A semantic video object is initialized with human assistance in a key frame. The video object is then tracked and segmented automatically in the following frames. A new active contour model, VSnakes, is introduced as a segmentation method in this framework. The active contour energy is defined so as to reflect the energy difference between two contours instead of the energy of a single contour. Multiple-resolution wavelet decomposition is applied in generating the edge energy of the image frame. Contour relaxation is used to deal with the object deformation frame by frame, and the Viterbi algorithm is used to update the contour path during contour relaxation. Compared to the original snakes algorithm, semiautomatic video object segmentation with the VSnakes algorithm resulted in improved performance in terms of video object shape distortion (1.4% versus 2.9% in one experiment), which suggests that it could be a useful tool in many content-based video applications, e.g., MPEG-4 video object generation and medical imaging.

This paper presents an algorithm for automatic segmentation of moving objects in video based on spatiotemporal visual saliency and an active contour model. Our algorithm exploits the visual saliency and motion information to build a spatiotemporal visual saliency map used to extract a moving region of interest. This region is used to automatically provide the seeds for the convex active contour (CAC) model to segment the moving object accurately. The experiments show a good performance of our algorithm for moving object segmentation in video without user interaction, especially on the SegTrack dataset.

Video object segmentation and tracking using region-based statistics

GPU-Accelerated Video Background Subtraction Using Gabor Detector

In this paper, an unsupervised video object (VO) segmentation and tracking algorithm is proposed based on an adaptable neural-network architecture. The proposed scheme comprises: 1) a VO tracking module and 2) an initial VO estimation module. Object tracking is handled as a classification problem and implemented through an adaptive network classifier, which provides better results compared to conventional motion-based tracking algorithms. Network adaptation is accomplished through an efficient and cost effective weight updating algorithm, providing a minimum degradation of the previous network knowledge and taking into account the current content conditions. A retraining set is constructed and used for this purpose based on initial VO estimation results. Two different scenarios are investigated. The first concerns extraction of human entities in video conferencing applications, while the second exploits depth information to identify generic VOs in stereoscopic video sequences. Human face/ body detection based on Gaussian distributions is accomplished in the first scenario, while segmentation fusion is obtained using color and depth information in the second scenario. A decision mechanism is also incorporated to detect time instances for weight updating. Experimental results and comparisons indicate the good performance of the proposed scheme even in sequences with complicated content (object bending, occlusion).