Bag Of Features Research Articles

A Hybrid Bags-of-Feature model for Sports Scene Classification

In this paper, we propose an effective framework for sports scene classification by using a Hybrid Bags-of-Feature model. The Bag-of-Feature (BoF) model is a methodology which represents an image based on the histogram of visual codewords. With its successful performance, the BoF model has been widely exploited in computer vision such as scene classification and object recognition. Unlike traditional BoF models which resort to a single feature descriptor such as SIFT and thus exploit a single codebook, the proposed Hybrid BoF model employs two different types of codebooks; SIFT feature and modified LBP feature. The LBP descriptor is capable of capturing micro patterns of the image, hence it is suitable for texture classification. In contrast, Gradient based feature descriptors such as SIFT have been proven to be effective for object recognition. More specifically, foreground objects (or regions containing distinct silhouettes) are encoded based on the SIFT descriptor and the remaining regions such as sky, lawn or floor of the gymnasium are encoded based on the LBP descriptor. To build a criterion for descriptor selection, we introduce a saliency pyramid based on the Phase Fourier Transform (PFT). The proposed method has been extensively tested, and experimental results show that the proposed framework is effective for sports scene classification compared to other various state-of-the-art methods.

Adaptive Euclidean maps for histograms: generalized Aitchison embeddings

Learning distances that are specifically designed to compare histograms in the probability simplex has recently attracted the attention of the machine learning community. Learning such distances is important because most machine learning problems involve bags of features rather than simple vectors. Ample empirical evidence suggests that the Euclidean distance in general and Mahalanobis metric learning in particular may not be suitable to quantify distances between points in the simplex. We propose in this paper a new contribution to address this problem by generalizing a family of embeddings proposed by Aitchison (J R Stat Soc 44:139---177, 1982) to map the probability simplex onto a suitable Euclidean space. We provide algorithms to estimate the parameters of such maps by building on previous work on metric learning approaches. The criterion we study is not convex, and we consider alternating optimization schemes as well as accelerated gradient descent approaches. These algorithms lead to representations that outperform alternative approaches to compare histograms in a variety of contexts.

Supervised learning of bag‐of‐features shape descriptors using sparse coding

AbstractWe present a method for supervised learning of shape descriptors for shape retrieval applications. Many content‐based shape retrieval approaches follow the bag‐of‐features (BoF) paradigm commonly used in text and image retrieval by first computing local shape descriptors, and then representing them in a ‘geometric dictionary’ using vector quantization. A major drawback of such approaches is that the dictionary is constructed in an unsupervised manner using clustering, unaware of the last stage of the process (pooling of the local descriptors into a BoF, and comparison of the latter using some metric). In this paper, we replace the clustering with dictionary learning, where every atom acts as a feature, followed by sparse coding and pooling to get the final BoF descriptor. Both the dictionary and the sparse codes can be learned in the supervised regime via bi‐level optimization using a task‐specific objective that promotes invariance desired in the specific application. We show significant performance improvement on several standard shape retrieval benchmarks.

Motion boundary based sampling and 3D co-occurrence descriptors for action recognition

Recent studies witness the success of Bag-of-Features (BoF) frameworks for video based human action recognition. The detection and description of local interest regions are two fundamental problems in BoF framework. In this paper, we propose a motion boundary based sampling strategy and spatial-temporal (3D) co-occurrence descriptors for action video representation and recognition. Our sampling strategy is partly inspired by the recent success of dense trajectory (DT) based features [Wang et al., 2013] for action recognition. Compared with DT, we densely sample spatial-temporal cuboids along a motion boundary which can greatly reduce the number of valid trajectories and preserve the discriminative power. Moreover, we develop a set of 3D co-occurrence descriptors which take account of the spatial-temporal context within local cuboids and deliver rich information for recognition. Furthermore, we decompose each 3D co-occurrence descriptor at pixel level and bin level and integrate the decomposed components with a multi-channel framework, which can improve the performance significantly. To evaluate the proposed methods, we conduct extensive experiments on three benchmarks including KTH, YouTube and HMDB51. The results show that our sampling strategy significantly reduces the computational cost of point tracking without degrading performance. Meanwhile, we achieve superior performance than the state-of-the-art methods. We report 95.6% on KTH, 87.6% on YouTube and 51.8% on HMDB51.

A food recognition system for diabetic patients based on an optimized bag-of-features model.

Computer vision-based food recognition could be used to estimate a meal's carbohydrate content for diabetic patients. This study proposes a methodology for automatic food recognition, based on the bag-of-features (BoF) model. An extensive technical investigation was conducted for the identification and optimization of the best performing components involved in the BoF architecture, as well as the estimation of the corresponding parameters. For the design and evaluation of the prototype system, a visual dataset with nearly 5000 food images was created and organized into 11 classes. The optimized system computes dense local features, using the scale-invariant feature transform on the HSV color space, builds a visual dictionary of 10000 visual words by using the hierarchical k-means clustering and finally classifies the food images with a linear support vector machine classifier. The system achieved classification accuracy of the order of 78%, thus proving the feasibility of the proposed approach in a very challenging image dataset.

Salient object detection and classification for stereoscopic images

Stereoscopic images have become more and more prevalent following the rapid advances in 3D capturing and display techniques. However, there has been little research on visual content analysis for stereoscopic images. In this paper, we address the challenging problem of object detection and classification for stereoscopic images. An iterative method that can mutually boost salient object detection and object classification is proposed for stereoscopic images. This method includes two steps. In the first step, a 3D saliency detection method, which includes the contrastive and occlusion cues contained in each stereoscopic image pair along with the discriminative features provided by the SVM classifier, is proposed to localize object of interest in the stereoscopic images. In the second step, the bag of word features of foreground and background is pooled by using the localization information, and then is applied to train the SVM classifier. Each of the two steps benefits from the gradual improvement result in the other, no matter in the training or the testing process. To evaluate the performance of our approach, a 6-object class dataset of stereoscopic images real objects viewed under general lighting conditions, poses and viewpoints is set up. Our experimental results on the dataset, for object localization and object classification, demonstrate the effectiveness of the method.

Soft-assigned bag of features for object tracking

Hard assignment-based bag of features (BoF) representation inevitably brings in quantization errors, which may lead to inaccuracy, even failure in object tracking. In this paper, we propose a novel soft-assigned BoF tracking approach, in which soft assignment is utilized to improve the robustness and discrimination of BoF representation. After labeling the tracked target, we first randomly sample the circle patches with adaptive size within and outside the labeled target, extract the local features from the patches, and construct the codebooks by k-means clustering. When tracking in a new frame, we generate the BoF representation of each candidate target, and select the most similar candidate target in the previous tracked result based on BoF representation. To improve tracking performance, we also continuously update the codebooks and refine the tracking results. Experiments show that our approach outperforms the state-of-the-art tracking methods under complex tracking conditions.

On-device mobile visual location recognition by using panoramic images and compressed sensing based visual descriptors.

Mobile Visual Location Recognition (MVLR) has attracted a lot of researchers' attention in the past few years. Existing MVLR applications commonly use Query-by-Example (QBE) based image retrieval principle to fulfill the location recognition task. However, the QBE framework is not reliable enough due to the variations in the capture conditions and viewpoint changes between the query image and the database images. To solve the above problem, we make following contributions to the design of a panorama based on-device MVLR system. Firstly, we design a heading (from digital compass) aware BOF (Bag-of-features) model to generate the descriptors of panoramic images. Our approach fully considers the characteristics of the panoramic images and can facilitate the panorama based on-device MVLR to a large degree. Secondly, to search high dimensional visual descriptors directly on mobile devices, we propose an effective bilinear compressed sensing based encoding method. While being fast and accurate enough for on-device implementation, our algorithm can also reduce the memory usage of projection matrix significantly. Thirdly, we also release a panoramas database as well as a set of test panoramic quires which can be used as a new benchmark to facilitate further research in the area. Experimental results prove the effectiveness of the proposed methods for on-device MVLR applications.

Spatial adjacent bag of features with multiple superpixels for object segmentation and classification

In the paper we present a new Spatial Adjacent Bag of Features (SABOF) model, in which the spatial information is effectively integrated into the traditional BOF model to enhance the scene and object recognition performance. The SABOF model chooses the frequency of each keyword and the largest frequency of its neighboring pairs to construct the feature histogram. Using the feature histogram whose dimension is only twice larger than that of the original BOF model, the SABOF model drastically enhances the discrimination performance. Combining the Superpixel Adjacent Histogram (SAH) Fulkerson et al., 2009 [12] with multiple segmentations Pantofaru et al., 2008 [33] and Russell et al., 2006 [36], the SABOF method effectively deals with the segmentation and classification of objects with different sizes. Changing the segmentation scale parameter to obtain multiple superpixel segmentations and correspondingly adjusting the neighbor parameters of the SAH method multiple classifiers are trained so that, the SABOF method can fuse multiple results of these classifiers to obtain better classification performance than any single classifier. The superpixel-based conditional random field (CRF) is used to further improve the classification performance. The experimental results of scene classification and of object recognition and localization on classical data sets demonstrate the performance of the proposed model and algorithm.

Hierarchical feature coding for image classification

Feature coding and pooling are two critical stages in the widely used Bag-of-Features (BOF) framework in image classification. After coding, each local feature formulates its representation by the visual codewords. However, the two-dimensional feature-code layout is transformed to a one-dimensional codeword representation after pooling. The property for each local feature is ignored and the whole representation is tightly coupled. To resolve this problem, we propose a hierarchical feature coding approach which regards each feature-code representation as a high level feature. Codeword learning, coding and pooling are also applied to these new features, and thus a high level representation of the image is obtained. Experiments on different datasets validate our analysis and demonstrate that the new representation is more discriminative than that in the previous BOF framework. Moreover, we show that various kinds of traditional feature coding algorithms can be easily embedded into our framework to achieve better performance.

On-Vehicle Video Localization Technique based on Video Search using Real Data on the Web

Recently, the mounting of on-vehicle camera is increasing to general cars. Because of this, some users start to upload the on-vehicle videos to web. So that, a number of on-vehicle videos are available nowadays. In this paper, in order to localize car, we propose the efficient matching method for such on-vehicle videos using Temporal Height Image (THI), Affine SIFT and Bag of Feature. THI retains information of relative building height from temporal image sequence. Then we extract robust features from the THI by using Affine SIFT. We realize efficient matching by expressing their features using Bag of features. We conducted experiments to show the efficiency of the proposed method by real image sequences of the city.

Spatially regularized and locality-constrained linear coding for human action recognition

To reduce quantization error, preserve the manifold of local features, distinguish the ambiguous features, and model the spatial configuration of features for Bag-of-Features (BoF) model-based human action recognition, a novel feature coding method called spatially regularized and locality-constrained linear coding (SLLC) is proposed. The spatial regularization and locality constraint are involved in the feature coding phase to model the spatial configuration of features and preserve their nonlinear manifold. The action recognition experimental results on benchmark datasets show that SLLC achieves better performance than the state-of-the-art feature coding methods such as soft vector quantization, sparse coding, and locality-constrained linear coding.

HUMAN ACTION RECOGNITION USING LOCAL SPACE TIME FEATURES AND ADABOOST SVM

Human action recognition has a wide range of promising applications like video surveillance, intelligent interface, and video retrieval. The objective of this project is to recognize and annotate different human activities present in digital videos taken from both constrained and unconstrained environment. This work extended the use of the techniques existing in object recognition in 2D images to video sequences by determining the Spatio-temporal Interest points using the extension of Harris operator in the time dimension. Features descriptors are computed on the cuboids around these interest points and further they are clustered and bag of features is built .SVM is used to classify the different classes of action present in the video. The recognition rate is further improved by using Adaboost SVM wherein number of weak classifier is weighted to form a strong classifier. The result shows that the proposed method using adaboost SVM classifier, the mean accuracy rate of recognition of KTH dataset is 89.13%.

Kernelized pyramid nearest-neighbor search for object categorization

Nearest-neighbor-based image classification has drawn considerable attention in the past several years thanks to its simplicity and efficiency. Recently, a Kernelized version of Naive-Bayes Nearest-Neighbor (KNBNN) approach has been proposed to combine Nearest-Neighbor-based approaches with other bag-of-feature (BoF) based kernels. However, similar to an orderless BoF image representation, the KNBNN ignores global geometric correspondence. In this paper, our contributions are threefolded. First, we present a technique to exploit the global geometric correspondence in a kernelized NBNN classifier framework. We divide an image into increasingly fine sub-regions like the spatial pyramid matching (SPM) approach; Second, we introduce a pyramid nearest-neighbor kernel by measuring the local similarity in each pyramid window. Third, for better calibrating the outputs of each window, we fit a sigmoid function to add posterior probability to its SVM outputs, and then weight these outputs of all windows. The sigmoid parameters and weight values are learned in a class-dependent and window-dependent manner. By doing so, we learn a class-specific geometric correspondence. Finally, the proposed approach is evaluated on two public datasets: Scene-15 and Caltech-101. We reach 85.2 % recognition rate on Scene-15 and 73.3 % on Caltech-101 only using single descriptor. The experimental results show that our approach significantly outperforms existing techniques.

Efficient BOF Generation and Compression for On-Device Mobile Visual Location Recognition

Existing mobile visual location recognition (MVLR) applications typically rely on bag-of-features (BOF) representation, which shows superior performance in retrieval accuracy. However, although the BOF framework is promising, it is not compact enough for on-device MVLR. The authors have made two contributions to the design of a BOF-based on-device MVLR system. First, to generate BOF descriptors, they propose a memory-efficient approximate nearest-neighbor search algorithm by combining residual vector quantization (RVQ) and tree-structured RVQ (TSRVQ). Second, they implemented a GPS-based and heading-aware RankBoost algorithm to reduce the dimensionality of the BOF descriptors. The authors evaluate the effectiveness of the proposed algorithms on an HTC mobile phone. Their work applies to on-device MVLR in city-scale workspaces.

MULTI-INSTANCE DICTIONARY LEARNING FOR DETECTING ABNORMAL EVENTS IN SURVEILLANCE VIDEOS

In this paper, a novel method termed Multi-Instance Dictionary Learning (MIDL) is presented for detecting abnormal events in crowded video scenes. With respect to multi-instance learning, each event (video clip) in videos is modeled as a bag containing several sub-events (local observations); while each sub-event is regarded as an instance. The MIDL jointly learns a dictionary for sparse representations of sub-events (instances) and multi-instance classifiers for classifying events into normal or abnormal. We further adopt three different multi-instance models, yielding the Max-Pooling-based MIDL (MP-MIDL), Instance-based MIDL (Inst-MIDL) and Bag-based MIDL (Bag-MIDL), for detecting both global and local abnormalities. The MP-MIDL classifies observed events by using bag features extracted via max-pooling over sparse representations. The Inst-MIDL and Bag-MIDL classify observed events by the predicted values of corresponding instances. The proposed MIDL is evaluated and compared with the state-of-the-art methods for abnormal event detection on the UMN (for global abnormalities) and the UCSD (for local abnormalities) datasets and results show that the proposed MP-MIDL and Bag-MIDL achieve either comparable or improved detection performances. The proposed MIDL method is also compared with other multi-instance learning methods on the task and superior results are obtained by the MP-MIDL scheme.

Image Registration based on Support Vector Machine for Tampering Localization

technologies are available on web. In the era of internet communication, systems should able to protect content such as pictures, videos against malicious modifications during their transmission. One of the important problems addressed in this is the authentication of the image received in a Communication. Tampering detection has significance in authentication of image. This paper presents support vector machine (SVM) based tampering detection system. In this a robust alignment (registration) method is proposed which makes use of an image hash component based on the Bag of Features (BOF) paradigm to localize the tampering. These BOF are clustered for effective image alignment. The support vector machine is optimal partitioning based linear classifier and at least theoretically better other classifier because only small numbers of classes required during classification SVM. The proposed signature is attached to the image before transmission and then analyzed at destination to recover the geometric transformations which have been applied to the received image. A block-wise tampering detection which uses histograms of oriented gradients (HOG) presentation is proposed. The proposed approach obtains better margin in providing an overall enhanced performance by reducing the training time while maintaining the accuracy.

Multiple-window Bag of Features for Road Environment Recognition

The idea of Bag of Features (BoF) is recently often employed for general object recognition. But, as it does not take positional relations of detected features into account, the recognition rate is still not very high for practical use. This paper proposes a method of describing the feature of an object by the BoF representation which considers positional information of the features. Although the original BoF representation is applied to an entire image, the proposed method employs multiple windows on an image. The BoF representation is applied to each of the windows to represent an object in the image interested for recognition. The performance of the proposed method is shown experimentally.

Context-aware and locality-constrained coding for image categorization.

Improving the coding strategy for BOF (Bag-of-Features) based feature design has drawn increasing attention in recent image categorization works. However, the ambiguity in coding procedure still impedes its further development. In this paper, we introduce a context-aware and locality-constrained Coding (CALC) approach with context information for describing objects in a discriminative way. It is generally achieved by learning a word-to-word cooccurrence prior to imposing context information over locality-constrained coding. Firstly, the local context of each category is evaluated by learning a word-to-word cooccurrence matrix representing the spatial distribution of local features in neighbor region. Then, the learned cooccurrence matrix is used for measuring the context distance between local features and code words. Finally, a coding strategy simultaneously considers locality in feature space and context space, while introducing the weight of feature is proposed. This novel coding strategy not only semantically preserves the information in coding, but also has the ability to alleviate the noise distortion of each class. Extensive experiments on several available datasets (Scene-15, Caltech101, and Caltech256) are conducted to validate the superiority of our algorithm by comparing it with baselines and recent published methods. Experimental results show that our method significantly improves the performance of baselines and achieves comparable and even better performance with the state of the arts.

Weighted feature trajectories and concatenated bag-of-features for action recognition

Key-point trajectory based approaches to recognizing human actions in realistic videos have recently shown promising results. However, their coverage of the entire actor is not sufficient for describing human actions, and the trajectories often disappear due to partial occlusions. In this paper, we propose a novel approach based on weighted feature trajectories and concatenated bag-of-features (BOF) using weighted spatio-temporal descriptors to overcome these limitations. To capture spatio-temporal regions of interest (ROI), we first build scale-invariant feature transform (SIFT) trajectories based on matching SIFT descriptors. Then, we extract particle trajectories around the SIFT trajectories using dense optical flow to retain ROI subsidiarily and to give a sound coverage of the actors. For balancing the leverages between SIFT and particle trajectories, a weighting scheme is presented according to the accumulated global distribution of feature points. Furthermore, we propose the concatenated two distinct SIFT and particle BOFs with weighted spatio-temporal descriptors for action recognition. Experimental results demonstrate the effectiveness of this approach in challenging real-world action datasets such as KTH, UCF sports, and TV human interaction.