Bag Of Visual Words Framework Research Articles

Feature fusion method using BoVW framework for enhancing image retrieval

The bag-of-visual words (BoVW) has been applied to myriad of recognition problems in computer vision such as object recognition, scene classification and image retrieval due to its scalability and high precision. However, their performance is subservient in certain datasets, especially in natural image datasets, mainly due to the lack of consideration of image cues such as colour, texture etc. which are not prime features while computing invariant descriptors, on which BoVW models are generally built on. Hence, this study describes a multi-cue fusion approach for BoVW framework, exploiting both early and late fusion methods, to improve the retrieval performance, mainly in natural image datasets. For this, a composite edge and colour descriptor is proposed to describe the local regions of the image along with the invariant feature descriptor Speeded Up Robust Features (SURF). Independent vocabularies are built based on these descriptors and images in the dataset are encoded to form two histograms using the respective vocabularies. The histograms are further fused to characterise the image. The retrieval is carried out by matching the histograms. Experimental results show that significant increment in the average precision can be attained by combining the proposed descriptor with invariant descriptors.

Action recognition in poor-quality spectator crowd videos using head distribution-based person segmentation

Despite a big volume of research on action recognition, little attention has been given to individual action recognition in poor-quality spectator crowd scenes. It is an important scenario, because most of the surveillance systems generate poor-quality videos, though current state-of-the-art methods may not be effectively applicable. Therefore recognizing actions performed by individuals in poor-quality spectator crowd scenes is an unsolved problem. In such cases, the main challenge is localizing person proposals for each actor in the crowd. This challenge becomes more difficult when occlusion is severe. In this work, we propose a novel approach to find person proposals in poor-quality spectator crowds using crowd-based constraints. First, we define persons in the crowd by using efficient person head detectors. We exploit person head size to estimate the person bounding box using linear regression. Then, we use distribution of heads in the crowd image to estimate more accurate person proposals. Motion trajectories are independently computed in the video without considering persons and then assigned to each person based on a novel distance measure computed between the trajectory and the person proposal. The set of trajectories and associated motion and texture-based features in overlapped time windows are used to compute the final feature vector. For each time window using early information fusion in the bag of visual-words framework, cumulative feature vectors are computed encoding action information. Experiments are performed on a publicly available real-world spectator crowd dataset containing as many as 150 actors performing multiple actions at the same time. Our experiments have demonstrated excellent performance of the proposed technique.

Segmentation Free Word Spotting for Handwritten Documents Using Bag of Visual Words Based on Co-HOG Descriptor

In this article, the authors propose a segmentation-free word spotting in handwritten document images using a Bag of Visual Words (BoVW) framework based on the co-occurrence histogram of oriented gradient (Co-HOG) descriptor. Initially, the handwritten document is represented using visual word vectors which are obtained based on the frequency of occurrence of Co-HOG descriptor within local patches of the document. The visual word representation vector does not consider their spatial location and spatial information helps to determine a location exclusively with visual information when the different location can be perceived as the same. Hence, to add spatial distribution information of visual words into the unstructured BoVW framework, the authors adopted spatial pyramid matching (SPM) technique. The performance of the proposed method evaluated using popular datasets and it is confirmed that the authors' method outperforms existing segmentation free word spotting techniques.

Violent scene detection algorithm based on kernel extreme learning machine and three-dimensional histograms of gradient orientation

Most existing feature descriptors for video have limited representation ability. In order to improve the recognition accuracy of method for detecting the videos that include violent scenes and take advantage of the logical structure of video sequences, a novel feature constructing approach based on three dimensional histograms of gradient orientation (HOG3D), the Bag of Visual Words (BoVW) model, and feature pooling technology is proposed. This approach, combined with kernel extreme learning machine (KELM), can be used to detect violent scene. First, the HOG3D feature is extracted on the block level for video, and then the K-Means clustering algorithm is implemented to generate visual words. Then, the bag of visual words framework is used for the quantization of feature. And the feature pooling technology is operated to generate a feature vector for an entire video segment, and feature vectors of training data and testing data were used separately to train the model and evaluate the performance of the proposed approach. The experimental results showed that the proposed feature descriptor had good representation and generalization abilities. The proposed approach is efficient for violent scene detection, and the accuracy matches the best result on Hockey dataset, and it outperforms state-of-the-art on Movies.

Image based classification of slums, built-up and non-built-up areas in Kalyan and Bangalore, India

ABSTRACT Slums, characterized by sub-standard housing conditions, are a common in fast growing Asian cities. However, reliable and up-to-date information on their locations and development dynamics is scarce. Despite numerous studies, the task of delineating slum areas remains a challenge and no general agreement exists about the most suitable method for detecting or assessing detection performance. In this paper, standard computer vision methods – Bag of Visual Words framework and Speeded-Up Robust Features have been applied for image-based classification of slum and non-slum areas in Kalyan and Bangalore, India, using very high resolution RGB images. To delineate slum areas, image segmentation is performed as pixel-level classification for three classes: Slums, Built-up and Non-Built-up. For each of the three classes, image tiles were randomly selected using ground truth observations. A multi-class support vector machine classifier has been trained on 80% of the tiles and the remaining 20% were used for testing. The final image segmentation has been obtained by classification of every 10th pixel followed by a majority filtering assigning classes to all remaining pixels. The results demonstrate the ability of the method to map slums with very different visual characteristics in two very different Indian cities.

Boosting image classification through semantic attention filtering strategies

Saliency Maps, frequently used to highlight significant information, can be combined with other paradigms, such as Bag of Visual Words (BoVW), to improve image description when the saliency regions correspond closely with the objects of interest. In this paper, we present three attention filtering strategies based on their saliency map that improve image classification using the BoVW framework, Spatial Pyramid Matching (SPM) and Convolutional Neural Networks (CNN) features. Firstly, we demonstrate how the blurring factor used in the Hou’s image signature algorithm determines what information remains and impacts to the obtained accuracy in image classification. Next, we propose AutoBlur, a simple but effective approach to automatically select this factor. Then, based on AutoBlur, we introduce two variants of our approach SARF (Semantic Attention Region Filtering), to semantically remove non-relevant regions through a Mean Shift segmentation. The first one is based on the intersection of the Hou’s image attention areas with its Mean Shift segmentation, while the second one discards regions using a key point voting system that relies on the Euclidean distance. The experiments carried out showed that the methods of Semantic Attention Filtering that we are proposing could be successfully used with both BoVW, SPM and CNN’s in most of the evaluated situations. In the five datasets assessed, all the three proposed methods outperform the baseline when using BoVWs in almost every case. For Spatial Pyramid Matching, the behaviour is similar, finding that the baseline is superior to our proposals in only one of the datasets used. In the case of CNN’s, our filtering proposal outperforms the baseline in two datasets, being very similar to it in the other cases.

Bag of Visual Words for Word Spotting in Handwritten Documents Based on Curvature Features

In this paper, we present a segmentation-based word spotting method for handwritten documents using Bag of Visual Words (BoVW) framework based on curvature features. The BoVW based word spotting methods extract SIFT or SURF features at each keypoint using fixed sized window. The drawbacks of these techniques are that they are memory intensive; the window size cannot be adapted to the length of the query and requires alignment between the keypoint sets. In order to overcome the drawbacks of SIFT or SURF local features based existing methods, we proposed to extract curvature feature at each keypoint of word image in BoVW framework. The curvature feature is scalar value describes the geometrical shape of the strokes and requires less memory space to store. The proposed method is evaluated using mean Average Precision metric through experimentation conducted on popular datasets such as GW, IAM and Bentham datasets. The yielded performances confirmed that our method outperforms existing word spotting techniques.

Land-use scene classification: a comparative study on bag of visual word framework

With successful launch of high spatial resolution (HSR) sensors, highly detailed spatial information is provided for remote sensing research. This improvement has allowed researchers to monitor environmental changes on a small spatial scale. However traditional pixel-based classification approaches are not able to interpret high spatial resolution remote sensing imagery effectively. Bag of visual words (BoVW) framework, on the other hand, is becoming one of the most popular approaches to validate the performance of remote sensing image datasets. While pixel-based approaches may not fully describe very high-resolution remote sensing images, BoVW model is narrowing the gap between low-level features and high-level semantic features by generating an intermediate description of image features. This paper presents a comparative study to evaluate the potential of using different coding approaches of BoVW model to solve the land-use scene classification problem. Initially, this work summarizes different configurations of BoVW framework in coding and clustering. Later, we perform an extensive evaluation of BoVW on land-use scene classification and retrieval. Finally we draw several conclusions regarding different coding strategies of BoVW, codebook size and number of training images. The approach is validated on two commonly used datasets in remote sensing, UC Merced a 21-class land-use dataset and RSDataset a 19-class satellite scene dataset.

Optimum Pipeline for Visual Terrain Classification Using Improved Bag of Visual Words and Fusion Methods

We propose an optimum pipeline and develop the hybrid representation to produce an effective and efficient visual terrain classification system. The bag of visual words (BOVW) framework has emerged as a promising approach and effective paradigm for visual terrain classification. The method includes four main steps: (1) feature extraction, (2) codebook generation, (3) feature coding, and (4) pooling and normalization. Recent researches have primarily focused on feature extraction in the development of new handcrafted descriptors that are specific to the visual terrain. However, the effects of other steps on visual terrain classification are still unknown. At the same time, fusion methods are often used to boost classification performance by exploring the complementarity of diverse features. We provide a comprehensive study of all steps in the BOVW framework and different fusion methods for visual terrain classification. Then, multiple approaches in each step and their effects are explored on the visual terrain dataset. Finally, the feature preprocessing technique, improved BOVW framework, and fusion method are used to construct an optimum pipeline for visual terrain classification. The hybrid representation developed by the optimum pipeline performs effectively and rapidly for visual terrain classification in the terrain dataset, outperforming those current methods. Furthermore, it is robust to diverse noises and illumination alterations.

Bag-of-Spatial Words(BoSW)Framework for Predicting SAR Image Registration in Real Time Applications

SAR Image registration is a precursor for several remote sensing applications, which need precise spatial transformation between the real time moving image and fixed off-line image. In such applications, the processing time in finding whether the moving images can be registered with fixed image constitute an overhead. Hence we have approached the problem by trying to predict if the given SAR images can be registered or not even without registering them. The proposed image registration approach incorporates a classifier into the standard pipeline of feature based image registration. The attributes for the classifier model are derived from fusing the spatial parameters of the feature detector to the descriptor vector in bag of visual words framework.

From pixels to gestures: learning visual representations for human analysis in color and depth data sequences

The visual analysis of humans from images is an important topic of interest due to its relevance to many computer vision applications likepedestrian detection, monitoring and surveillance, human-computer interaction, e-health or content-based image retrieval, among others.In this dissertation we are interested in learning different visual representations of the human body that are helpful for the visual analysis of humans in images and video sequences. To that end, we analyze both RGB and depth image modalities and address the problem from three different research lines, at different levels of abstraction; from pixels to gestures: human segmentation, human pose estimation and gesture recognition.First, we show how binary segmentation (object vs. background) of the human body in image sequences is helpful to remove all the background clutter present in the scene. The presented method, based on Graph cuts optimization, enforces spatio-temporal consistency of the produced segmentation masks among consecutive frames. Secondly, we present a framework for multi-label segmentation for obtaining much more detailed segmentation masks: instead of just obtaining a binary representation separating the human body from the background, finer segmentation masks can be obtained separating the different body parts.At a higher level of abstraction, we aim for a simpler yet descriptive representation of the human body. Human pose estimation methods usually rely on skeletal models of the human body, formed by segments (or rectangles) that represent the body limbs, appropriately connected following the kinematic constraints of the human body. In practice, such skeletal models must fulfill some constraints in order to allow for efficient inference, while actually limiting the expressiveness of the model. In order to cope with this, we introduce a top-down approach for predicting the position of the body parts in the model, using a mid-level part representation based on Poselets.Finally, we propose a framework for gesture recognition based on the bag of visual words framework. We leverage the benefits of RGB and depth image modalities by combining modality-specific visual vocabularies in a late fusion fashion. A new rotation-variant depth descriptor is presented, yielding better results than other state-of-the-art descriptors. Moreover, spatio-temporal pyramids are used to encode rough spatial and temporal structure. In addition, we present a probabilistic reformulation of Dynamic Time Warping for gesture segmentation in video sequences. A Gaussian-based probabilistic model of a gesture is learnt, implicitly encoding possible deformations in both spatial and time domains.

Clasificación de vehículos basada en bolsa de palabras visuales y usando imágenes de rango

Los descriptores de características en 3D extraídos de nubes de puntos se han convertido en una fuente deinformación promisoria para muchas aplicaciones. Estas aplicaciones incluyen reconocimiento de objetos o formas,modelamiento de edificaciones, modelamiento de estructuras civiles, sistemas de navegación autónoma, etc.Considerando estas tendencias, este artículo presenta un sistema de clasificación de vehículos, basado en bolsade palabras, el cual extrae descriptores de características de imágenes de rango capturadas usando un láser SICKLMS200. Nuestro enfoque usa información visual para estimar la velocidad de los vehículos usando un filtro deKalman, esta estimación de la velocidad es usada para registrar los datos del láser y construir la nube de puntosde la escena. En este trabajo, un dataset fue capturado a partir de la nube de puntos del vehículo, su informaciónvisual, la estimación de la velocidad del vehículo y una etiqueta de la clase de vehículo. Usando este dataset, variosdescriptores de 3D fueron probados y árboles KD fueron empleados para acelerar el proceso de clasificación, esteúltimo realizado con la Bolsa de Palabras. En consecuencia, nuestro enfoque puede clasificar hasta nueve diferentesclases de vehículos; el desempeño del clasificador fue medido usando curvas tipo Precision –Recall.

Informative patches sampling for image classification by utilizing bottom-up and top-down information

In image classification based on bag of visual words framework, image patches used for creating image representations affect the classification performance significantly. However, currently, patches are sampled mainly based on processing low-level image information or just extracted regularly or randomly. These methods are not effective, because patches extracted through these approaches are not necessarily discriminative for image categorization. In this paper, we propose to utilize both bottom-up information through processing low-level image information and top-down information through exploring statistical properties of training image grids to extract image patches. In the proposed work, an input image is divided into regular grids, each of which is evaluated based on its bottom-up information and/or top-down information. Subsequently, every grid is assigned a saliency value based on its evaluation result, so that a saliency map can be created for the image. Finally, patch sampling from the input image is performed on the basis of the obtained saliency map. Furthermore, we propose a method to fuse these two kinds of information. The proposed methods are evaluated on both object categories and scene categories. Experiment results demonstrate their effectiveness.

Writer identification in handwritten musical scores with bags of notes

Writer Identification is an important task for the automatic processing of documents. However, the identification of the writer in graphical documents is still challenging. In this work, we adapt the Bag of Visual Words framework to the task of writer identification in handwritten musical scores. A vanilla implementation of this method already performs comparably to the state-of-the-art. Furthermore, we analyze the effect of two improvements of the representation: a Bhattacharyya embedding, which improves the results at virtually no extra cost, and a Fisher Vector representation that very significantly improves the results at the cost of a more complex and costly representation. Experimental evaluation shows results more than 20 points above the state-of-the-art in a new, challenging dataset.

Medical Image Categorization and Retrieval System in Radiology Using Bag of Visual Words Framework

In This work we present an efficient image categorization and retrieval system applied to Image Clef 2009 medical image retrieval task. In this task we have presented methodology is based on local patch representation of the image content and a bag-of-features approach for defining image categories, with a kernel based SVM classifier. Two main tasks are addressed: First organ identification task; second the detection and identification of pathologies, i.e. shifting from the organ level to pathology level analysis. We used a large generic archive of 12,000 radiographs (IRMA) to tune the system parameters. We demonstrate automated organ detection on the IRMA collection as well as the generalization to a new data collection. We submitted one run, using support-vector-machines trained on the visual word histograms in multiple scales. We proposed system was helped to find discriminating orientation and body regions in X-ray images also organ-level discrimination we show an application to pathology level categorization of chest X-ray data. Results indicate detection of pathology at a sensitivity of 88.4% and a specificity of 81%. This is first step towards similarity-based medical image categorization that has a major clinical importance in computer-assisted diagnostics. It can identify suspicious pathological X-rays and alert the referring clinicians to potential emergencies. Overall it is hoped that the development of such systems will contribute to the improvement of safety and quality of medical services