Bag Of Visual Words Research Articles

Partial-duplicate image retrieval using spatial and visual contextual clues

The traditional bag-of-visual-words (BOW) model quantifies the local features to the visual words to achieve efficient content-based image retrieval. However, since it causes considerable quantisation error and ignores the spatial relationships between visual words, the accuracy of partial-duplicate image retrieval based on BOW model is limited. In order to reduce the quantisation error and improve the discriminability of visual words, many partial-duplicate image retrieval methods have been proposed, which make use of the advantages of the geometric clues between visual words. In this paper, we propose a novel partial-duplicate scheme by using both spatial and visual contextual clues for removing the false matches effectively, which not only encodes the relationships of orientation, distance and dominant orientation between the referential visual word and its context, but also takes the colour information between visual words into consideration. Experimental results reveal that our proposed algorithm achieves superior performance to the state-of-the-art methods for partial-duplicate image retrieval.

Local Learning With Deep and Handcrafted Features for Facial Expression Recognition

We present an approach that combines automatic features learned by convolutional neural networks (CNN) and handcrafted features computed by the bag-of-visual-words (BOVW) model in order to achieve state-of-the-art results in facial expression recognition. To obtain automatic features, we experiment with multiple CNN architectures, pre-trained models and training procedures, e.g. Dense-Sparse-Dense. After fusing the two types of features, we employ a local learning framework to predict the class label for each test image. The local learning framework is based on three steps. First, a k-nearest neighbors model is applied in order to select the nearest training samples for an input test image. Second, a one-versus-all Support Vector Machines (SVM) classifier is trained on the selected training samples. Finally, the SVM classifier is used to predict the class label only for the test image it was trained for. Although we have used local learning in combination with handcrafted features in our previous work, to the best of our knowledge, local learning has never been employed in combination with deep features. The experiments on the 2013 Facial Expression Recognition (FER) Challenge data set, the FER+ data set and the AffectNet data set demonstrate that our approach achieves state-of-the-art results. With a top accuracy of 75.42% on FER 2013, 87.76% on the FER+, 59.58% on AffectNet 8-way classification and 63.31% on AffectNet 7-way classification, we surpass the state-of-the-art methods by more than 1% on all data sets.

A New Intelligent Fault Diagnosis Method of Rotating Machinery under Varying‐Speed Conditions Using Infrared Thermography

A novel systematic framework, infrared thermography‐ (IRT‐) based method, for rotating machinery fault diagnosis under nonstationary running conditions is presented in this paper. In this framework, IRT technique is first applied to obtain the thermograph. Then, the fault features are extracted using bag‐of‐visual‐word (BoVW) from the IRT images. In the end, support vector machine (SVM) is utilized to automatically identify the fault patterns of rotating machinery. The effectiveness of proposed method is evaluated using lab experimental signal of rotating machinery. The diagnosis results show that the IRT‐based method has certain advantages in classification rotating machinery faults under nonstationary running conditions compared with the traditional vibration‐based method.

Loop Closure Detection Via Maximization of Mutual Information

An image can be described in terms of appearance frequency of visual words. This representation is implemented in bag-of-visual-words (BoVW)-based loop closure detection for its efficiency and effectiveness. However, traditional BoVW-based approaches are strongly affected by false positive loops due to scene ambiguity caused by redundant words in the vocabulary and fail to detect bidirectional loops in monocular mode. Aiming at overcoming these problems, we propose a novel vocabulary construction algorithm named hierarchical sequential information bottleneck (HsIB) by leveraging the maximization of mutual information (MMI) mechanism. First, feature descriptors are extracted from training images for visual vocabulary construction. Second, HsIB extracts discriminative yet informative visual words through the MMI mechanism in vocabulary construction, which treats feature descriptors clustering as a process of data compression. Finally, the clustering process reaches a tradeoff between compactness and discrimination and improves the performance of traditional BoVW-based loop closure detection. The proposed method is compared with state-of-the-art methods on publicly available datasets. We also create a challenging dataset to further evaluate the performance of HsIB on bidirectional loops. To the best of our knowledge, we are the first to implement information bottleneck (IB) method in visual-SLAM (vSLAM) loop closure detection, and we obtain impressive results.

A text-Image feature mapping algorithm based on transfer learning

Abstract The traditional uniform distribution algorithm does not filter the image data when extracting the approximate features of text-image data under the event, so the similarity between the image data and the text is low, which leads to low accuracy of the algorithm. This paper proposes a text-image feature mapping algorithm based on transfer learning. The existing data is filtered by ‘clustering technology’ to obtain similar data with the target data. The significant text features are calculated through the latent Dirichlet allocation (LDA) model and information gain based on Gibbs sampling. Bag of visual word (BOVW) model and Naive Bayesian method are used to model image data. With the help of the text-image co-occurrence data in the same event, the text feature distribution is mapped to the image feature space, and the feature distribution of image data under the same event is approximated. Experimental results show that the proposed algorithm can obtain the feature distribution of image data under different events, and the average cosine similarity is as high as 92%, the average dispersion is as low as 0.06%, and the accuracy of the algorithm is high.

Violent scene detection algorithm based on kernel extreme learning machine and three-dimensional histograms of gradient orientation

Most existing feature descriptors for video have limited representation ability. In order to improve the recognition accuracy of method for detecting the videos that include violent scenes and take advantage of the logical structure of video sequences, a novel feature constructing approach based on three dimensional histograms of gradient orientation (HOG3D), the Bag of Visual Words (BoVW) model, and feature pooling technology is proposed. This approach, combined with kernel extreme learning machine (KELM), can be used to detect violent scene. First, the HOG3D feature is extracted on the block level for video, and then the K-Means clustering algorithm is implemented to generate visual words. Then, the bag of visual words framework is used for the quantization of feature. And the feature pooling technology is operated to generate a feature vector for an entire video segment, and feature vectors of training data and testing data were used separately to train the model and evaluate the performance of the proposed approach. The experimental results showed that the proposed feature descriptor had good representation and generalization abilities. The proposed approach is efficient for violent scene detection, and the accuracy matches the best result on Hockey dataset, and it outperforms state-of-the-art on Movies.

Novel computer‐aided diagnosis of lung cancer using bag of visual words to achieve high accuracy rates

Lung Cancer is considered as one of the deadliest diseases among other lung disorders. Several researchers studied and investigated about classifying the lung cancer images and their approaches do not provide better accuracy. The objective of this paper is mainly focused on detecting the cancerous region and classifying the lung cancer CT images. The proposed work plays a significant role in detecting the types of cancerous region in the lung CT images and helps radiologists to enhance the diagnosis in medical care of patients. It's also a less time consuming process in classifying the lung CT images. Bag of Visual Words (BoVW) is a novel classification technique which is applied on lung CT images to classify the cluster of begnin or malignant lung cancer in different steps. With the help of MATLAB 2016, the simulation results are obtained. The achieved accuracy of BoVW classifier is 96% which is higher than other classifiers applied on lung cancer CT images. This classifier highly reduces the computational complexity. This CAD system is very beneficial to radiologists and oncologists in cancer research centres or any cancer institutions. By applying other segmentation or novel classification techniques may improve the automatic diagnosis of lung cancer images.

Fusing depth and colour information for human action recognition

In recent years, human action recognition systems have been increasingly developed to support a wide range of application areas, such as surveillance, behaviour analysis, security, and many others. In particular, data fusion approaches that use depth and colour information (i.e., RGB-D data) seem to be particularly promising for recognizing large classes of human actions with a high level of accuracy. Anyway, existing data fusion approaches are mainly based on feature fusion strategies, which tend to suffer of some limitations, including the difficult of combining different feature types and the management of missing information. To address the two problems just reported, we propose an RGB-D data based human action recognition system supported by a decision fusion strategy. The system, starting from the well-known Joint Directors of Laboratories (JDL) data fusion model, analyses human actions separately for each channel (i.e., depth and colour). The actions are modelled as a sum of visual words by using the traditional Bag-of-Visual-Words (BoVW) model. Subsequently, on each channel, these actions are classified by using a multi-class Support Vector Machine (SVM) classifier. Finally, the classification results are fused by a Naive Bayes Combination (NBC) method. The effectiveness of the proposed system has been proven on the basis of three public datasets: UTKinect-Action3D, CAD-60, and LIRIS Human Activities. Experimental results, compared with key works of the current state-of-the-art, have shown that what we propose can be considered a concrete contribute to the action recognition field.

A Novel Discriminating and Relative Global Spatial Image Representation with Applications in CBIR

The requirement for effective image search, which motivates the use of Content-Based Image Retrieval (CBIR) and the search of similar multimedia contents on the basis of user query, remains an open research problem for computer vision applications. The application domains for Bag of Visual Words (BoVW) based image representations are object recognition, image classification and content-based image analysis. Interest point detectors are quantized in the feature space and the final histogram or image signature do not retain any detail about co-occurrences of features in the 2D image space. This spatial information is crucial, as it adversely affects the performance of an image classification-based model. The most notable contribution in this context is Spatial Pyramid Matching (SPM), which captures the absolute spatial distribution of visual words. However, SPM is sensitive to image transformations such as rotation, flipping and translation. When images are not well-aligned, SPM may lose its discriminative power. This paper introduces a novel approach to encoding the relative spatial information for histogram-based representation of the BoVW model. This is established by computing the global geometric relationship between pairs of identical visual words with respect to the centroid of an image. The proposed research is evaluated by using five different datasets. Comprehensive experiments demonstrate the robustness of the proposed image representation as compared to the state-of-the-art methods in terms of precision and recall values.

Video Scene Detection Using Compact Bag of Visual Word Models

Video segmentation into shots is the first step for video indexing and searching. Videos shots are mostly very small in duration and do not give meaningful insight of the visual contents. However, grouping of shots based on similar visual contents gives a better understanding of the video scene; grouping of similar shots is known as scene boundary detection or video segmentation into scenes. In this paper, we propose a model for video segmentation into visual scenes using bag of visual word (BoVW) model. Initially, the video is divided into the shots which are later represented by a set of key frames. Key frames are further represented by BoVW feature vectors which are quite short and compact compared to classical BoVW model implementations. Two variations of BoVW model are used: (1) classical BoVW model and (2) Vector of Linearly Aggregated Descriptors (VLAD) which is an extension of classical BoVW model. The similarity of the shots is computed by the distances between their key frames feature vectors within the sliding window of length L, rather comparing each shot with very long lists of shots which has been previously practiced, and the value of L is 4. Experiments on cinematic and drama videos show the effectiveness of our proposed framework. The BoVW is 25000-dimensional vector and VLAD is only 2048-dimensional vector in the proposed model. The BoVW achieves 0.90 segmentation accuracy, whereas VLAD achieves 0.83.

Tea Garden Detection from High-Resolution Imagery Using a Scene-Based Framework

Tea cultivation has a long history in China, and it is one of the pillar industries of the Chinese agricultural economy. It is therefore necessary to map tea gardens for their ongoing management. However, the previous studies have relied on fieldwork to achieve this task, which is time-consuming. In this paper, we propose a framework to map tea gardens using high-resolution remotely sensed imagery, including three scene-based methods: the bag-of-visual-words (BOVW) model, supervised latent Dirichlet allocation (sLDA), and the unsupervised convolutional neural network (UCNN). These methods can develop direct and holistic semantic representations for tea garden scenes composed of multiple sub-objects, thus they are more suitable than the traditional pixel-based or object-based methods, which focus on the local characteristics of pixels or objects. In the experiments undertaken in this study, the three different methods were tested on four datasets from Longyan (Oolong tea), Hangzhou (Longjing tea), and Puer (Puer tea). All the methods achieved a good performance, both quantitatively and visually, and the UCNN outperformed the other methods. Moreover, it was found that the addition of textural features improved the accuracy of the BOVW and sLDA models, but had no effect on the UCNN.

Bag of local features for person re-identification on large-scale datasets

In recent years, large-scale person re-identification has attracted a lot of attention from video surveillance. Usual approaches addressing this task either learn the effective feature embeddings or design the learning architectures to obtain discriminative metrics. Most of them only focus on improving the accuracy of recognition but neglect retrieval efficiency. To improve the accuracy and efficiency of person re-identification simultaneously, an accurate and fast method is proposed based on the bag of visual words (BoVW) model, which has widely been applied in image retrieval. A bag of local features is developed to simplify feature representation for person re-identification. Cross-view dictionary learning is used to eliminate the redundancy of these local features. These local features consist of scale invariant feature transform and local maximal occurrence representation (LOMO) that are invariant in scale and color, respectively. Finally, integrated BoVW histograms are obtained, which encode the images by k-means clustering. Experiments conducted on the CUHK03, Market1501, and MARS datasets show that the proposed method performs favorably against existing approaches.

Content-based medical image retrieval by spatial matching of visual words

Content-Based Image Retrieval (CBIR) systems have recently emerged as one of the most promising and best image retrieval paradigms. To pacify the semantic gap associated with CBIR systems, the Bag of Visual Words (BoVW) techniques are now increasingly used. However, existing BoVW techniques fail to capture the location information of visual words effectively. This paper proposes an unsupervised Content-Based Medical Image Retrieval (CBMIR) framework based on the spatial matching of the visual words. The proposed method efficiently computes the spatial similarity of visual words using a novel similarity measure called the Skip Similarity Index. Experiments on three large medical datasets reveal promising results. The location-based correlation of visual words assists in more accurate and efficient retrieval of anatomically diverse and multimodal medical images than the state-of-the-art CBMIR systems.

CONFIRM – Clustering of noisy form images using robust matching

Identifying the type of a scanned form image greatly facilitates automated processing, including field segmentation and field recognition. Contrary to most prior work, we focus on unsupervised type identification, where the possible form types for a given collection are not known apriori. Our target domain is noisy collections of form images that contain structurally similar, yet objectively different, form types, which are challenging to differentiate in an unsupervised setting. This work presents a novel algorithm: CONFIRM (Clustering Of Noisy Form Images using Robust Matching), which simultaneously discovers the set of form types in a collection and assigns a type to each form. CONFIRM matches type-set text and rule lines between forms to create collection-specific features, which we show outperform the Bag of Visual Word (BoVW) approach employed by the current state-of-the-art in form image clustering. CONFIRM scales well to large document collections with a bootstrap clustering process, in which only a small subset of the data is clustered directly, and the rest of the data is assigned to clusters in linear time. We show that CONFIRM reduces cluster impurity on average by 44% compared to the state-of-the art on 5 collections of historical forms that contain structurally similar form types.

A signal-to-image transformation approach for EEG and MEG signal classification

The classification of magnetic signals has become one of the challenging research problems in brain computer interfaces (BCIs). Magnetic signals, as measured with electroencephalography (EEG) and magnetoencephalography (MEG), contain lots of additional information on the bioelectrical activity of the brain. In this paper, we propose a simple transformation method that utilises signal-to-image conversion. This conversion is a kind of finite amplitude frequency transformation based on the changing points of the signals. In other words, arbitrary time domain signals are converted to two-dimensional finite images, which are then used in the classification of the signals. In feature extraction, the Harris corner detector and scale-invariant feature transform are combined with the bag of visual words, and these features are then classified by using the k-nearest neighbour algorithm. To confirm the validity of the proposed method, experiments are conducted on the BCI Competition 2003 Datasets Ia and BCI Competition 2008 Dataset III. The classification accuracy of the proposed method is over 96.21% for Dataset Ia and 78.99% for Dataset III. It is apparent from the results that the EEG and MEG signals are quite successfully classified by employing the proposed method.

Weed Mapping with UAS Imagery and a Bag of Visual Words Based Image Classifier

Weed detection with aerial images is a great challenge to generate field maps for site-specific plant protection application. The requirements might be met with low altitude flights of unmanned aerial vehicles (UAV), to provide adequate ground resolutions for differentiating even single weeds accurately. The following study proposed and tested an image classifier based on a Bag of Visual Words (BoVW) framework for mapping weed species, using a small unmanned aircraft system (UAS) with a commercial camera on board, at low flying altitudes. The image classifier was trained with support vector machines after building a visual dictionary of local features from many collected UAS images. A window-based processing of the models was used for mapping the weed occurrences in the UAS imagery. The UAS flight campaign was carried out over a weed infested wheat field, and images were acquired between a 1 and 6 m flight altitude. From the UAS images, 25,452 weed plants were annotated on species level, along with wheat and soil as background classes for training and validation of the models. The results showed that the BoVW model allowed the discrimination of single plants with high accuracy for Matricaria recutita L. (88.60%), Papaver rhoeas L. (89.08%), Viola arvensis M. (87.93%), and winter wheat (94.09%), within the generated maps. Regarding site specific weed control, the classified UAS images would enable the selection of the right herbicide based on the distribution of the predicted weed species.

Novel Distributional Visual-Feature Representations for image classification

The Bag-of-Visual-Words (BoVW) representation is a well known strategy to approach many computer vision problems. The idea behind BoVW is similar to the Bag-of-Words (BoW) used in text mining tasks: to build word histograms to represent documents. Regarding computer vision, most of the research has been devoted to obtain better visual words, rather than in improving the final representation. This is somewhat surprising, as there are many alternative ways of improving the BoW representation within the text mining community that can be applied in computer vision as well. This paper aims at evaluating the usefulness of Distributional Term Representations (DTRs) for image classification. DTRs represent instances by exploiting statistics of feature occurrences and co-occurrences along the dataset. We focus in the suitability and effectiveness of using well-known DTRs in different image collections. Furthermore, we devise two novel distributional strategies that learn appropriated groups of images to compute better suited distributional features. We report experimental results in several image datasets showing the effectiveness of the proposed DTRs over BoVW and other methods in the literature including deep learning based strategies. In particular we show the effectiveness of the proposed representations on image collections from narrow domains, where target categories are subclasses of a more general class (e.g., subclasses of birds, aircrafts, or dogs).

A Filter of Minhash for Image Similarity Measures

Image similarity measures play an important role in nearest neighbor search and duplicate detection for large-scale image datasets. Recently, Minwise Hashing (or Minhash) and its related hashing algorithms have achieved great performances in large-scale image retrieval systems. However, there are a large number of comparisons for image pairs in these applications, which may spend a lot of computation time and affect the performance. In order to quickly obtain the pairwise images that theirs similarities are higher than the specific thresholdT(e.g., 0.5), we propose a dynamic threshold filter of Minwise Hashing for image similarity measures. It greatly reduces the calculation time by terminating the unnecessary comparisons in advance. We also find that the filter can be extended to other hashing algorithms, on when the estimator satisfies the binomial distribution, such as b-Bit Minwise Hashing, One Permutation Hashing, etc. In this pager, we use the Bag-of-Visual-Words (BoVW) model based on the Scale Invariant Feature Transform (SIFT) to represent the image features. We have proved that the filter is correct and effective through the experiment on real image datasets.

RIHOG-BoVWs for Rotation-invariant Human Detection

Rotation-invariant human detection is of vital significance to many applications, including victim detection via a daylight camera on an unmanned aerial vehicle (UAV). With this problem in mind, we propose a powerful rotation-invariant object-level description method, i.e., RIHOG-BoVWs, which is achieved by local-level feature calculation and high-level semantic feature extraction. In local-level feature calculation, we consider Rotation-invariant Histograms of Oriented Gradients (RIHOG) as the local descriptor and project gradient information into the frequency domain, where the rotation-invariant representation vector is constructed. In high-level semantic feature extraction, Bag of Visual Words (BoVWs) model is employed to achieve global description from local features without considering their spatial structures. Experimentally, we first confirm the sufficient discrimination power of the RIHOG-BoVWs on the public Freestyle Motocross dataset, and then demonstrate the high performance of RIHOG-BoVWs on a victim dataset which has varied backgrounds and body postures.

Robust 3D Action Recognition Through Sampling Local Appearances and Global Distributions

Three-dimensional (3-D) action recognition has broad applications in human–computer interaction and intelligent surveillance. However, recognizing similar actions remains challenging since previous literature fails to capture motion and shape cues effectively from noisy depth data. In this paper, we propose a novel two-layer Bag-of-Visual-Words (BoVW) model, which suppresses the noise disturbances and jointly encodes both motion and shape cues. First, background clutter is removed by a background modeling method that is designed for depth data. Then, motion and shape cues are jointly used to generate robust and distinctive spatial-temporal interest points (STIPs): motion-based STIPs and shape-based STIPs. In the first layer of our model, a multiscale 3-D local steering kernel descriptor is proposed to describe local appearances of cuboids around motion-based STIPs. In the second layer, a spatial-temporal vector descriptor is proposed to describe the spatial-temporal distributions of shape-based STIPs. Using the BoVW model, motion and shape cues are combined to form a fused action representation. Our model performs favorably compared with common STIP detection and description methods. Thorough experiments verify that our model is effective in distinguishing similar actions and robust to background clutter, partial occlusions and pepper noise.