Covariance Descriptor Research Articles

Covariance descriptor based on bio-inspired features for person re-identification and face verification

Avoiding the use of complicated pre-processing steps such as accurate face and body part segmentation or image normalization, this paper proposes a novel face/person image representation which can properly handle background and illumination variations. Denoted as gBiCov, this representation relies on the combination of Biologically Inspired Features (BIF) and Covariance descriptors [1]. More precisely, gBiCov is obtained by computing and encoding the difference between BIF features at different scales. The distance between two persons can then be efficiently measured by computing the Euclidean distance of their signatures, avoiding some time consuming operations in Riemannian manifold required by the use of Covariance descriptors. In addition, the recently proposed KISSME framework [2] is adopted to learn a metric adapted to the representation. To show the effectiveness of gBiCov, experiments are conducted on three person re-identification tasks (VIPeR, i-LIDS and ETHZ) and one face verification task (LFW), on which competitive results are obtained. As an example, the matching rate at rank 1 on the VIPeR dataset is of 31.11%, improving the best previously published result by more than 10.

Tensor Sparse Coding for Positive Definite Matrices

In recent years, there has been extensive research on sparse representation of vector-valued signals. In the matrix case, the data points are merely vectorized and treated as vectors thereafter (for example, image patches). However, this approach cannot be used for all matrices, as it may destroy the inherent structure of the data. Symmetric positive definite (SPD) matrices constitute one such class of signals, where their implicit structure of positive eigenvalues is lost upon vectorization. This paper proposes a novel sparse coding technique for positive definite matrices, which respects the structure of the Riemannian manifold and preserves the positivity of their eigenvalues, without resorting to vectorization. Synthetic and real-world computer vision experiments with region covariance descriptors demonstrate the need for and the applicability of the new sparse coding model. This work serves to bridge the gap between the sparse modeling paradigm and the space of positive definite matrices.

Modeling Geometric-Temporal Context With Directional Pyramid Co-Occurrence for Action Recognition.

In this paper, we present a new geometric-temporal representation for visual action recognition based on local spatio-temporal features. First, we propose a modified covariance descriptor under the log-Euclidean Riemannian metric to represent the spatio-temporal cuboids detected in the video sequences. Compared with previously proposed covariance descriptors, our descriptor can be measured and clustered in Euclidian space. Second, to capture the geometric-temporal contextual information, we construct a directional pyramid co-occurrence matrix (DPCM) to describe the spatio-temporal distribution of the vector-quantized local feature descriptors extracted from a video. DPCM characterizes the co-occurrence statistics of local features as well as the spatio-temporal positional relationships among the concurrent features. These statistics provide strong descriptive power for action recognition. To use DPCM for action recognition, we propose a directional pyramid co-occurrence matching kernel to measure the similarity of videos. The proposed method achieves the state-of-the-art performance and improves on the recognition performance of the bag-of-visual-words (BOVWs) models by a large margin on six public data sets. For example, on the KTH data set, it achieves 98.78% accuracy while the BOVW approach only achieves 88.06%. On both Weizmann and UCF CIL data sets, the highest possible accuracy of 100% is achieved.

Action recognition using global spatio-temporal features derived from sparse representations

Recognizing actions is one of the important challenges in computer vision with respect to video data, with applications to surveillance, diagnostics of mental disorders, and video retrieval. Compared to other data modalities such as documents and images, processing video data demands orders of magnitude higher computational and storage resources. One way to alleviate this difficulty is to focus the computations to informative (salient) regions of the video. In this paper, we propose a novel global spatio-temporal self-similarity measure to score saliency using the ideas of dictionary learning and sparse coding. In contrast to existing methods that use local spatio-temporal feature detectors along with descriptors (such as HOG, HOG3D, and HOF), dictionary learning helps consider the saliency in a global setting (on the entire video) in a computationally efficient way. We consider only a small percentage of the most salient (least self-similar) regions found using our algorithm, over which spatio-temporal descriptors such as HOG and region covariance descriptors are computed. The ensemble of such block descriptors in a bag-of-features framework provides a holistic description of the motion sequence which can be used in a classification setting. Experiments on several benchmark datasets in video based action classification demonstrate that our approach performs competitively to the state of the art.

一种基于协方差描述子的场景分类算法

Scene classification is a hot topic in computer vision.Under the premise of image segmentation,a novel scene classification algorithm is proposed,which combines pixel location,color characteristics,direction features and local texture features to form the covariance descriptor.To avoid computing tedious distance measure in Riemannian space,the covariance descriptor is converted into sigma-point representation,where scene describing and SVM based training can be completed in Euclidian space.The performance of the novel algorithm is compared with some of classical algorithms using SUN Database.Farther more,the robustness of the algorithm is validated with noise appended data samples.The results show that the proposed algorithm not only has advantages on computation time and classification performance,but also has good robustness to scene noise.

Abrupt motion tracking using a visual saliency embedded particle filter

Abrupt motion is a significant challenge that commonly causes traditional tracking methods to fail. This paper presents an improved visual saliency model and integrates it to a particle filter tracker to solve this problem. Once the target is lost, our algorithm recovers tracking by detecting the target region from salient regions, which are obtained in the saliency map of current frame. In addition, to strengthen the saliency of target region, the target model is used as a prior knowledge to calculate a weight set which is utilized to construct our improved saliency map adaptively. Furthermore, we adopt the covariance descriptor as the appearance model to describe the object more accurately. Compared with several other tracking algorithms, the experimental results demonstrate that our method is more robust in dealing with various types of abrupt motion scenarios.

Color Invariants for Person Reidentification

We revisit the problem of specific object recognition using color distributions. In some applications--such as specific person identification--it is highly likely that the color distributions will be multimodal and hence contain a special structure. Although the color distribution changes under different lighting conditions, some aspects of its structure turn out to be invariants. We refer to this structure as an intradistribution structure, and show that it is invariant under a wide range of imaging conditions while being discriminative enough to be practical. Our signature uses shape context descriptors to represent the intradistribution structure. Assuming the widely used diagonal model, we validate that our signature is invariant under certain illumination changes. Experimentally, we use color information as the only cue to obtain good recognition performance on publicly available databases covering both indoor and outdoor conditions. Combining our approach with the complementary covariance descriptor, we demonstrate results exceeding the state-of-the-art performance on the challenging VIPeR and CAVIAR4REID databases.

Miniature illustrations retrieval and innovative interaction for digital illuminated manuscripts

In this paper we propose a multimedia solution for the interactive exploration of illuminated manuscripts. We leveraged on the joint exploitation of content-based image retrieval and relevance feedback to provide an effective mechanism to navigate through the manuscript and add custom knowledge in the form of tags. The similarity retrieval between miniature illustrations is based on covariance descriptors, integrating color, spatial and gradient information. The proposed relevance feedback technique, namely Query Remapping Feature Space Warping, accounts for the user’s opinions by accordingly warping the data points. This is obtained by means of a remapping strategy (from the Riemannian space where covariance matrices lie, referring back to Euclidean space) useful to boost the retrieval performance. Experiments are reported to show the quality of the proposal. Moreover, the complete prototype with user interaction, as already showcased at museums and exhibitions, is presented.

Visual saliency estimation by nonlinearly integrating features using region covariances

To detect visually salient elements of complex natural scenes, computational bottom-up saliency models commonly examine several feature channels such as color and orientation in parallel. They compute a separate feature map for each channel and then linearly combine these maps to produce a master saliency map. However, only a few studies have investigated how different feature dimensions contribute to the overall visual saliency. We address this integration issue and propose to use covariance matrices of simple image features (known as region covariance descriptors in the computer vision community; Tuzel, Porikli, & Meer, 2006) as meta-features for saliency estimation. As low-dimensional representations of image patches, region covariances capture local image structures better than standard linear filters, but more importantly, they naturally provide nonlinear integration of different features by modeling their correlations. We also show that first-order statistics of features could be easily incorporated to the proposed approach to improve the performance. Our experimental evaluation on several benchmark data sets demonstrate that the proposed approach outperforms the state-of-art models on various tasks including prediction of human eye fixations, salient object detection, and image-retargeting.

Salient point region covariance descriptor for target tracking

Features extracted at salient points are used to construct a region covariance descriptor (RCD) for target tracking. In the classical approach, the RCD is computed by using the features at each pixel location, which increases the computational cost in many cases. This approach is redundant because image statistics do not change significantly between neighboring image pixels. Furthermore, this redundancy may decrease tracking accuracy while tracking large targets because statistics of flat regions dominate region covariance matrix. In the proposed approach, salient points are extracted via the Shi and Tomasi’s minimum eigenvalue method over a Hessian matrix, and the RCD features extracted only at these salient points are used in target tracking. Experimental results indicate that the salient point RCD scheme provides comparable and even better tracking results compared to a classical RCD-based approach, scale-invariant feature transform, and speeded-up robust features-based trackers while providing a computationally more efficient structure.

Image classification of human carcinoma cells using complex wavelet-based covariance descriptors.

Cancer cell lines are widely used for research purposes in laboratories all over the world. Computer-assisted classification of cancer cells can alleviate the burden of manual labeling and help cancer research. In this paper, we present a novel computerized method for cancer cell line image classification. The aim is to automatically classify 14 different classes of cell lines including 7 classes of breast and 7 classes of liver cancer cells. Microscopic images containing irregular carcinoma cell patterns are represented by subwindows which correspond to foreground pixels. For each subwindow, a covariance descriptor utilizing the dual-tree complex wavelet transform (DT-WT) coefficients and several morphological attributes are computed. Directionally selective DT-WT feature parameters are preferred primarily because of their ability to characterize edges at multiple orientations which is the characteristic feature of carcinoma cell line images. A Support Vector Machine (SVM) classifier with radial basis function (RBF) kernel is employed for final classification. Over a dataset of 840 images, we achieve an accuracy above 98%, which outperforms the classical covariance-based methods. The proposed system can be used as a reliable decision maker for laboratory studies. Our tool provides an automated, time- and cost-efficient analysis of cancer cell morphology to classify different cancer cell lines using image-processing techniques, which can be used as an alternative to the costly short tandem repeat (STR) analysis. The data set used in this manuscript is available as supplementary material through http://signal.ee.bilkent.edu.tr/cancerCellLineClassificationSampleImages.html.

A Novel Dynamic System in the Space of SPD Matrices with Applications to Appearance Tracking

In this paper, we address the problem of video tracking using covariance descriptors constructed from simple features extracted from the given image sequence. Theoretically, this can be posed as a tracking problem in the space of ($n \times n$) symmetric positive definite (SPD) matrices denoted by $P_n$. A novel probabilistic dynamic model in $P_n$ based on Riemannian geometry and probability theory is presented in conjunction with a geometric (intrinsic) recursive filter for tracking a time sequence of SPD matrix measurements in a Bayesian framework. This newly developed filtering method can be used for the covariance descriptor updating problem in covariance tracking, leading to new and efficient video tracking algorithms. To show the accuracy and efficiency of our tracker in comparison to the state-of-the-art, we present synthetic experiments on $P_n$ and several real data experiments for tracking in video sequences.

A Hierarchical Ship Detection Scheme for High-Resolution SAR Images

This paper presents a new hierarchical scheme for detecting ships from high-resolution synthetic aperture radar (SAR) images. The scheme consists of two stages: detection and discrimination. In the detection stage, the existing internal Hermitian product is extended to obtain a new detector. The new detector makes a combined use of the complex coherence among more than two subapertures and the intensity of each subaperture. When the subaperture number is increased, the target/clutter contrast is shown to be improved. Ship candidates are obtained by applying a threshold. Ship discrimination is performed by using one-class classification. The covariance descriptor, developed by Tuzel in 2006, is introduced to SAR ship discrimination as the feature. The traditional one-class quadratic discriminator is used as the discriminator. After this stage, most false alarms are rejected, and the real ship targets in the candidates are maintained. The effectiveness of the proposed scheme is verified using RADARSAT-2 data. Experimental results show that the proposed scheme can detect most ship targets in the image and few false alarms occur.

Vision-based behavior prediction of ball carrier in basketball matches

A new vision-based approach was presented for predicting the behavior of the ball carrier-shooting, passing and dribbling in basketball matches. It was proposed to recognize the ball carrier’s head pose by classifying its yaw angle to determine his vision range and the court situation of the sportsman within his vision range can be further learned. In basketball match videos characterized by cluttered background, fast motion of the sportsmen and low resolution of their head images, and the covariance descriptor, were adopted to fuse multiple visual features of the head region, which can be seen as a point on the Riemannian manifold and then mapped to the tangent space. Then, the classification of head yaw angle was directly completed in this space through the trained multiclass LogitBoost. In order to describe the court situation of all sportsmen within the ball carrier’s vision range, artificial potential field (APF)-based information was introduced. Finally, the behavior of the ball carrier-shooting, passing and dribbling, was predicted using radial basis function (RBF) neural network as the classifier. Experimental results show that the average prediction accuracy of the proposed method can reach 80% on the video recorded in basketball matches, which validates its effectiveness.

Real-Time Probabilistic Covariance Tracking With Efficient Model Update

The recently proposed covariance region descriptor has been proven robust and versatile for a modest computational cost. The covariance matrix enables efficient fusion of different types of features, where the spatial and statistical properties, as well as their correlation, are characterized. The similarity between two covariance descriptors is measured on Riemannian manifolds. Based on the same metric but with a probabilistic framework, we propose a novel tracking approach on Riemannian manifolds with a novel incremental covariance tensor learning (ICTL). To address the appearance variations, ICTL incrementally learns a low-dimensional covariance tensor representation and efficiently adapts online to appearance changes of the target with only O(1) computational complexity, resulting in a real-time performance. The covariance-based representation and the ICTL are then combined with the particle filter framework to allow better handling of background clutter, as well as the temporary occlusions. We test the proposed probabilistic ICTL tracker on numerous benchmark sequences involving different types of challenges including occlusions and variations in illumination, scale, and pose. The proposed approach demonstrates excellent real-time performance, both qualitatively and quantitatively, in comparison with several previously proposed trackers.

Classifier-based offline feature selection and evaluation for visual tracking of sea-surface and aerial targets

An offline feature selection and evaluation mechanism is used in order to develop a robust visual tracking scheme for sea-surface and aerial targets. The covariance descriptors, known to constitute an efficient signature set in object detection and classification problems, are used in the feature extraction phase of the proposed scheme. The performance of feature sets are compared using support vector machines, and those resulting in the highest detection performance are used in the covariance based tracker. The tracking performance is evaluated in different scenarios using different performance measures with respect to ground truth target positions. The proposed tracking scheme is observed to track sea-surface and aerial targets with plausible accuracies, and the results show that gradient-based features, together with the pixel locations and intensity values, provide robust target tracking in both surveillance scenarios. The performance of the proposed tracking strategy is also compared with some well-known trackers including correlation, Kanade-Lucas-Tomasi feature, and scale invariant feature transform-based trackers. Experimental results and observations show that the proposed target tracking scheme outperforms other trackers in both air and sea surveillance scenarios.

A fast multi-scale covariance descriptor for object re-identification

In many surveillance systems, there is a need to determine if a given object (person, group of persons, vehicle, …) has already been observed over a network of cameras. It is the object re-identification problem. Solving this problem involves matching observation of objects across disjoint camera views. Uncalibrated fixed or mobile cameras with non-overlapping field of view generate uncontrolled variation in view point, background and lighting. In such situations, a robust and invariant image description is required. A multi-scale covariance image descriptor and a quadtree based scheme are proposed to describe any object of interest. We describe a fast method for computation of multi-scale covariance descriptor. The descriptor is evaluated in person re-identification application using the VIPeR dataset. We show that the proposed multi-scale approach outperforms existing mono-scale image description methods.

An Approach to the Fast ROI Detection System using Region Descriptor and Background Modeling

In this paper, fast region of interests(ROI) detection system is proposed using background modeling technique and neural network in a video stream. Existing systems for tracking and detection on a video stream could generate many loads because of detecting a region of interests on entire region at every frame. To solve the problem, the proposed system is detecting a ROI on moving region using background modeling technique and neural network. First, the proposed system extracts features on ROI using covariance descriptor and is studied through a backpropagation (BP) neural network. And then the proposed system can abstract a moving region using background modeling technique and detect a ROI from the moving region using covariance descriptor and neural network. In this paper, we conduct a performance evaluation of a proposed system through experiments on detecting a license plate in a video stream on a road and prove a quality of a proposed system.

Contextual Information and Covariance Descriptors for People Surveillance: An Application for Safety of Construction Workers

In computer science, contextual information can be used both to reduce computations and to increase accuracy. This paper discusses how it can be exploited for people surveillance in very cluttered environments in terms of perspective (i.e., weak scene calibration) and appearance of the objects of interest (i.e., relevance feedback on the training of a classifier). These techniques are applied to a pedestrian detector that uses a LogitBoost classifier, appropriately modified to work with covariance descriptors which lie on Riemannian manifolds. On each detected pedestrian, a similar classifier is employed to obtain a precise localization of the head. Two novelties on the algorithms are proposed in this case: polar image transformations to better exploit the circular feature of the head appearance and multispectral image derivatives that catch not only luminance but also chrominance variations. The complete approach has been tested on the surveillance of a construction site to detect workers that do not wear the hard hat: in such scenarios, the complexity and dynamics are very high, making pedestrian detection a real challenge.

Covariance Tracking via Geometric Particle Filtering

Region covariance descriptor recently proposed has been approved robust and elegant to describe a region of interest, which has been applied to visual tracking. We develop a geometric method for visual tracking, in which region covariance is used to model objects appearance; then tracking is led by implementing the particle filter with the constraint that the system state lies in a low dimensional manifold: affine Lie group. The sequential Bayesian updating consists of drawing state samples while moving on the manifold geodesics; the region covariance is updated using a novel approach in a Riemannian space. Our main contribution is developing a general particle filtering-based racking algorithm that explicitly take the geometry of affine Lie groups into consideration in deriving the state equation on Lie groups. Theoretic analysis and experimental evaluations demonstrate the promise and effectiveness of the proposed tracking method.