Challenging Benchmark Video Research Articles

Automatic Video Object Segmentation Based on Visual and Motion Saliency

We present an approach to extract the salient object automatically in videos. Given an unannotated video sequence, the proposed method first computes the visual saliency to identify object-like regions in each frame based on the proposed weighted multiple manifold ranking algorithm. We then compute motion cues to estimate the motion saliency and localization prior. Finally, adopting a new energy function, we estimate a superpixel-level object labeling across all frames, where 1) the data term depends on the visual saliency and localization prior, and 2) the smoothness term depends on the constraints in time and space. Compared to the existing counterparts, the proposed approach automatically segments the persistent foreground object meanwhile preserving the potential shape. Experiments show its promising results on the challenging benchmark videos in comparison with the existing counterparts.

Online Visual Tracking of Weighted Multiple Instance Learning via Neutrosophic Similarity-Based Objectness Estimation

An online neutrosophic similarity-based objectness tracking with a weighted multiple instance learning algorithm (NeutWMIL) is proposed. Each training sample is extracted surrounding the object location, and the distribution of these samples is symmetric. To provide a more robust weight for each sample in the positive bag, the asymmetry of the importance of the samples is considered. The neutrosophic similarity-based objectness estimation with object properties (super straddling) is applied. The neutrosophic theory is a new branch of philosophy for dealing with incomplete, indeterminate, and inconsistent information. By considering the surrounding information of the object, a single valued neutrosophic set (SVNS)-based segmentation parameter selection method is proposed, to produce a well-built set of superpixels which can better explain the object area at each frame. Then, the intersection and shape-distance criteria are proposed for weighting each superpixel in the SVNS domain, mainly via three membership functions, T (truth), I (indeterminacy), and F (falsity), for each criterion. After filtering out the superpixels with low response, the newly defined neutrosophic weights are utilized for weighting each sample. Furthermore, the objectness estimation information is also applied for estimating and alleviating the problem of tracking drift. Experimental results on challenging benchmark video sequences reveal the superior performance of our algorithm when confronting appearance changes and background clutters.

Correlation filter‐based visual tracking via adaptive weighted CNN features fusion

Visual object tracking is an important and challenging task in computer vision. In this study, the authors propose a novel visual tracking approach by decomposing the tracking task into translation and scale estimation. In translation estimation, they employ multiple adaptive correlation filters with features of hierarchical convolutional neural networks (CNNs) to more accurately estimate the target location. To make full use of multi-level features from different CNN layers, they propose an adaptive weighted algorithm to fuse correlation response maps. In scale estimation, a one-dimensional correlation filter with histogram of oriented gradient (HOG) features is employed to estimate the scale variation. Extensive experimental results on 50 challenging benchmark video sequences demonstrate that the proposed algorithm outperforms state-of-the-art algorithms.

Visual Tracking via Adaptive Random Projection Based on Sub-Regions

This paper aims to track arbitrary single target object in a video sequence given its location in the first frame and no other information. In order to track the location and further reduce the influence of occlusion, a part-based appearance model is constructed with color, texture, and spatial structure features extracted in the compressed domain. Moreover, the confidence distributions of different sub-regions bring rich information of appearance change, which enables us to duly update the classifier parameters and to further improve the robustness and stability. In order to reduce the appearance change caused by scale interference, median flow tracking is employed to estimate the scale variation among consecutive frames. Extensive evaluations on challenging benchmark video sequences demonstrate that the proposed tracking algorithm outperforms several state-of-the-art methods in terms of success, precision, robustness, and stability.

How do deep convolutional features affect tracking performance: an experimental study

Visual tracking is an import topic in computer vision with many practical applications. Recently, deep learning methods have been introduced into the tracking community to improve tracking performance. How deep features affect tracking performance, however, has not been studied thoroughly. In this paper, we carry out an experimental study to deeply investigate the impact of convolutional features on tracking performance. We adopt the most influential and representative Convolutional Neural Network (CNN) models that are widely used in computer vision to equip our baseline tracking framework. Firstly, we carry out experiments on each CNN to reveal the relationship between tracking performance and different CNN layers. Secondly, we have a vertical comparison of tracking performance between different CNN models. In addition, we explore the effect of ensemble strategies of CNN features on tracking performance. Our work has got several valuable findings on the relationship between tracking performance and convolutional features. Based on our findings, we have derived a few useful guidelines for designing trackers with better performance. We have also developed a simple baseline tracker with the guidelines and it outperforms several state-of-the-art trackers very easily on challenging benchmark video sequences.

Online object tracking based on BLSTM-RNN with contextual-sequential labeling

Object context has been verified its significance for appearance modeling in different proposed tracking-by-detection approaches. Unfortunately, the restrictive representation of the target’s contextual relationship within spatial domain has intensively limited its utility with high-level classification strategies. By investigating the learning capability of long-term dependencies from sequential data, in this paper, we propose a novel appearance model by transforming the target contextual dependency into a semantic sequential representation. It can be effectively utilized by a recurrent neural network embedded with bidirectional long short-term memory cells for online tracking-by-learning. Based on the trained BLSTM-RNN model, a searching mechanism by labeling score is proposed to improve the tracking robustness. With the implied appearance variation by labeling, the proposed tracking method has demonstrated to outperform most of state-of-the-art trackers on challenging benchmark videos via a heuristic strategy for model updating.

Weakly supervised activity analysis with spatio-temporal localisation

In computer vision, an increasing number of weakly annotated videos have become available, due to the fact it is often difficult and time consuming to annotate all the details in the videos collected. Learning methods that analyse human activities in weakly annotated video data have gained great interest in recent years. They are categorised as “weakly supervised learning”, and usually form a multi-instance multi-label (MIML) learning problem. In addition to the commonly known difficulties of MIML learning, i.e. ambiguities in instances and labels, a weakly supervised method also has to cope with large data size, high dimensionality, and a large proportion of noisy examples usually found in video data. In this work, we propose a novel learning framework that iteratively optimises over a scalable MIML model and an instance selection process incorporating pairwise spatio-temporal smoothing during training. Such learned knowledge is then generalised to testing via a noise removal process based on the support vector data description algorithm. According to the experiments on three challenging benchmark video datasets, the proposed framework yields a more discriminative MIML model and less noisy training and testing data, and thus improves the system performance. It outperforms the state-of-the-art weakly supervised and even fully supervised approaches in the literature, in terms of annotating and detecting actions of a single person and interactions between a pair of people.

Robust object tracking based upon adaptive multi-cue integration for video surveillance

It was well argued in literature that integrating multi-cue increases accuracy and robustness of visual tracking. Although, multi-cues object tracking using singlemodal or multimodal was explored by some of researchers, it still remains an open challenge to fuse multi-cue from different modularity under dynamic environment conditions. The aim of this research paper is to introduce a novel multi-cue object tracking framework using particle filter. In particle filter framework, our approach integrates cues while evaluating each particle instead of primitive approach of deciding cues performance in current frame based upon either a few present particles or previous state particles. First, we model our multi-cue tracking framework using Shafer's model and multi-cue data is combined using Conjunctive combination rules. The partial and total conflict among cues at particle level is redistributed efficiently using Proportional Conflict Redistribution (PCR-5) rules. In proposed model, automatic suppression/boosting of particles along with online conflict resolving facilitate resampling process for efficiently handling of particle degeneracy. Most importantly, compared to other state-of-art trackers, our proposed algorithm can handle more efficiently various dynamic environments conditions such as partial or full occlusion, illumination changes, weather, and visibility. In this manuscript, we demonstrate our proposed adaptive multi-cue fusion model on challenging benchmark video and thermal sequences and compare tracking results of our tracker with state-of- art trackers.

Real-time tracking-by-learning with high-order regularization fusion for big video abstraction

Visual tracking is a key technique used by video abstraction to achieve efficient post-analysis for big video surveillance. In order to tackle the problem of constantly changing scenarios during online tracking, additional factors such as motion can be incorporated by utilizing a fusion strategy to improve the final performance. Unfortunately, straightforward output fusion is difficult to be synchronized due to the diversity in model regression. Therefore, a widely cited problem for learning based fusion is to incorporate regularizers for label assignment of unlabeled samples, which is one of the major research focuses on semi-supervised learning. In this paper, a novel tracking strategy based on semi-supervised learning with high order regularization fusion has been proposed. It employs two different types of regularizers to achieve more accurate label assignment based on kernelized confidence prediction and graph-based bi-directional trace from motion. The computation of the proposed tracker takes advantage of the unique feature of circulant matrix in Fourier domain and integral patterns, and thus can be readily implemented for real-time processing, even without any code optimization. Via a dynamic budget maintenance for model updating, the proposed tracking method demonstrated to outperform most state-of-art trackers on challenging benchmark videos with a fixed parameter configuration.

Visual tracking with multifeature joint sparse representation

We present a visual tracking method with feature fusion via joint sparse presentation. The proposed method describes each target candidate by combining different features and joint sparse representation for robustness in coefficient estimation. Then, we build a probabilistic observation model based on the approximation error between the recovered candidate image and the observed sample. Finally, this observation model is integrated with a stochastic affine motion model to form a particle filter framework for visual tracking. Furthermore, a dynamic and robust template update strategy is applied to adapt the appearance variations of the target and reduce the possibility of drifting. Quantitative evaluations on challenging benchmark video sequences demonstrate that the proposed method is effective and can perform favorably compared to several state-of-the-art methods.

Visual object tracking via online sparse instance learning

Sparse representation has been attracting much more attention in visual tracking. However most sparse representation based trackers only focus on how to model the target appearance and do not consider the learning of sparse representation when the training samples are imprecise, and hence may drift or fail in the challenging scene. In this paper, we present a novel online tracking algorithm. The tracker integrates the online multiple instance learning into the recent sparse representation scheme. For tracking, the integrated sparse representation combining texture, intensity and local spatial information is proposed to model the target. This representation takes both occlusion and appearance change into account. Then, an efficient online learning approach is proposed to select the most distinguishable features to separate the target from the background samples. In addition, the sparse representation is dynamically updated online with respect to the current context. Both qualitative and quantitative evaluations on challenging benchmark video sequences demonstrate that the proposed tracking algorithm performs favorably against several state-of-the-art methods.

Visual Tracking Based on Reversed Sparse Representation

In this paper, we propose a fast and robust tracking method based on reversed sparse representation. Be different from other sparse representation based visual tracking methods, the target template is sparsely represented by the candidate particles which are gotten by particle filter. In order to improve the robustness of the method, we use a target template set. Meanwhile, a two level competition mechanism is also introduced. In the first level, each target template is sparsely represented and all the candidate particles compete with each other by a similarity calculation, which is based on sparse coefficients. Then, the winners construct a target candidate set. In the second level, all the target candidates in the target candidate set compete with each other and the one which is the most similar to the template set is considered as the target. In addition, a template set update strategy is proposed to adapt the appearance variations of the target. Experimental results on challenging benchmark video sequences demonstrate that the proposed tracking algorithm performs favorably against several state-of-the-art methods.

Exploiting spatial redundancy of image sensor for motion robust rPPG.

Remote photoplethysmography (rPPG) techniques can measure cardiac activity by detecting pulse-induced color variations on human skin using an RGB camera. State-of-the-art rPPG methods are sensitive to subject body motions (e.g., motion-induced color distortions). This study proposes a novel framework to improve the motion robustness of rPPG. The basic idea of this paper originates from the observation that a camera can simultaneously sample multiple skin regions in parallel, and each of them can be treated as an independent sensor for pulse measurement. The spatial redundancy of an image sensor can thus be exploited to distinguish the pulse signal from motion-induced noise. To this end, the pixel-based rPPG sensors are constructed to estimate a robust pulse signal using motion-compensated pixel-to-pixel pulse extraction, spatial pruning, and temporal filtering. The evaluation of this strategy is not based on a full clinical trial, but on 36 challenging benchmark videos consisting of subjects that differ in gender, skin types, and performed motion categories. Experimental results show that the proposed method improves the SNR of the state-of-the-art rPPG technique from 3.34 to 6.76 dB, and the agreement ( ±1.96σ) with instantaneous reference pulse rate from 55% to 80% correct. ANOVA with post hoc comparison shows that the improvement on motion robustness is significant. The rPPG method developed in this study has a performance that is very close to that of the contact-based sensor under realistic situations, while its computational efficiency allows real-time processing on an off-the-shelf computer.

Structured partial least squares based appearance model for visual tracking

The appearance model is an important issue in the visual tracking. In this paper, we present a structured partial least squares (SPLS) based appearance model for object tracking which not only discriminates the target from its surrounding background but also able to tolerate its appearance variations. The target is represented as a low-dimensional feature vector, which is the structured combination of the subspaces learned by SPLS. In order to capture the variations in appearance during the tracking, we update the appearance model by taking the estimated class labels of the new targets obtained online into account. Furthermore, aiming to alleviate the drift problems, negative samples generated by disordering the structured appearance model are added in the training samples. Both qualitative and quantitative evaluations on challenging benchmark video sequences demonstrate that the proposed tracking algorithm performs favorably against several state-of-the-art methods.

Tracking and recognition face in videos with incremental local sparse representation model

This paper addresses the problem of tracking and recognizing faces via incremental local sparse representation. First a robust face tracking algorithm is proposed via employing local sparse appearance and covariance pooling method. In the following face recognition stage, with the employment of a novel template update strategy, which combines incremental subspace learning, our recognition algorithm adapts the template to appearance changes and reduces the influence of occlusion and illumination variation. This leads to a robust video-based face tracking and recognition with desirable performance. In the experiments, we test the quality of face recognition in real-world noisy videos on YouTube database, which includes 47 celebrities. Our proposed method produces a high face recognition rate at 95% of all videos. The proposed face tracking and recognition algorithms are also tested on a set of noisy videos under heavy occlusion and illumination variation. The tracking results on challenging benchmark videos demonstrate that the proposed tracking algorithm performs favorably against several state-of-the-art methods. In the case of the challenging dataset in which faces undergo occlusion and illumination variation, and tracking and recognition experiments under significant pose variation on the University of California, San Diego (Honda/UCSD) database, our proposed method also consistently demonstrates a high recognition rate.