Visual Codebook Research Articles

Shape classification using a new shape descriptor and multi-view learning

Shape classification is considered as a vital task in solving many computer vision problems. Different factors such as affine transformations, scaling, rotations, variation in perspective, noise and occlusion have made the shape classification problem to be a hard problem. This work investigates a new shape descriptor that extracts different features from each boundary pixel. This makes it to be more informative and discriminant in comparison with other descriptors. After feature extraction, the “bag-of-features (BoF)” model is employed to construct the final representation for each image. To enhance the functionality of the BoF model, a novel codebook generation approach is presented. The proposed approach tends to derive a more meaningful visual codebook. Consequently, the produced feature vectors can handle inter- and intra-class variations more effectively. Comprehensive experiments conducted on the various complicated shape datasets show the supremacy of our approach compared to other methods.

Toward Innovative Recognition of Handwritten Arabic Characters: A Hybrid Approach with SIFT, BoVW, and SVM classification

The goal of handwriting recognition has been a top priority for those who want to enter data into computer systems for more than thirty years. In several fields, the advent of handwriting recognition technology is highly anticipated. OCR technology has made it possible for computers to recognize characters as visual objects and collect data about their unique characteristics in recent years. In particular, several studies in this field have focused on Arabic writing. The use of machines to examine handwritten papers is the first step in the character identification process. The identification of specific Arabic characters is the main goal of this particular investigation. In computer vision, Arabic character recognition is very important since it's necessary to correctly recognize and classify Arabic letters and characters in manuscripts. In this research, an innovative approach based on identifying Arabic character characteristics using BoVW (bag of visual words) and SIFT (Scale Invariant Feature Transform) features is proposed. These features are clustered using k-means clustering to produce a dictionary. Following that, SVM (Support Vector Machine) is utilized to classify the word images in a visual codebook created using these terms. The proposed approach is an innovative method to deal with the difficulties associated with Arabic hand-writing recognition. The utilization of BoVW and SIFT features is expected to enhance the system's robustness in recognizing and classifying Arabic characters. The proposed approach will be experimentally evaluated using a dataset that includes a variety of Arabic characters written in various styles. The results of this study will offer important new perspectives on the effectiveness and practicality of the approach suggested

Multiple Distance-Based Coding: Toward Scalable Feature Matching for Large-Scale Web Image Search

For scalable feature matching in large-scale web image search, the bag-of-visual-words-based (BOW) approaches generally code local features as visual words to construct an inverted index file to match features efficiently. Both the popular feature coding techniques, i.e., K-means-based vector quantization and scalar quantization, directly quantize features to generate visual words. K-means-based vector quantization requires expensive visual codebook training, whereas scalar quantization leads to the miss of many matches due to the low stability of individual components of feature vectors. To address the above issues, we demonstrate that the corresponding sub-vectors of similar features generally have similar distances to multiple reference points in feature subspace and propose a multiple distance-based feature coding scheme for scalable feature matching. Specifically, based on the distances between the sub-vectors and multiple distinct reference points, we transform each feature to a set of feature codes, where one code is treated as a visual word required to construct the inverted index file whereas the others are embedded into the index file to further verify the feature matching based on the visual words. The proposed coding scheme does not need visual codebook training and shows desirable stability and discriminability. Moreover, in the matching verification, a feature-distance estimation method is proposed to estimate the Euclidean distances between features for an accurate matching verification. Extensive experimental results demonstrate the superiority of the proposed approach in comparison to the other approaches using recent feature quantization methods for large-scale web image search.

Image Retrieval using Hybrid Order Dither Block Truncation Code

A new approach is proposed to index images in database using features generated from the HODBTC compressed data stream. This indexing technique can be extended for CBIR. HODBTC compresses an image into a set of color quantizers and a bitmap image. The proposed image retrieval system generates two image features namely CCF and BPF from the minimum quantizer, maximum quantizer and bitmap image respectively by involving the visual codebook.

Bag-of-Visual-Words codebook generation using deep features for effective classification of imbalanced multi-class image datasets

Classification of imbalanced multi-class image datasets is a challenging problem in computer vision. Most of the real-world datasets are imbalanced in nature because of the uneven distribution of the samples in each class. The problem with an imbalanced dataset is that the minority class having a smaller number of instance samples is left undetected. Most of the traditional machine learning algorithms can detect the majority class efficiently but lag behind in the efficient detection of the minority class, which ultimately degrades the overall performance of the classification model. In this paper, we have proposed a novel combination of visual codebook generation using deep features with the non-linear Chi2 SVM classifier to tackle the imbalance problem that arises while dealing with multi-class image datasets. The low-level deep features are first extracted by transfer learning using the ResNet-50 pre-trained network, and clustered using k-means. The center of each cluster is a visual word in the codebook. Each image is then translated into a set of features called the Bag-of-Visual-Words (BOVW) derived from the histogram of visual words in the vocabulary. The non-linear Chi2 SVM classifier is found most optimal for classifying the ensuing features, as proved by a detailed empirical analysis. Hence with the right combination of learning tools, we are able to tackle classification of multi-class imbalanced image datasets in an effective manner. This is proved from the higher scores of accuracy, F1-score and AUC metrics in our experiments on two challenging multi-class datasets: Graz-02 and TF-Flowers, as compared to the state-of-the-art methods.

Adaptive bag‐of‐visual word modelling using stacked‐autoencoder and particle swarm optimisation for the unsupervised categorisation of images

The bag-of-visual words (BOVWs) have been recognised as an effective mean of representing images for image classification. However, its reliance on a visual codebook developed using handcrafted image feature extraction algorithms and vector quantisation via k -means clustering often results in significant computational overhead, and poor classification accuracies. Therefore, this study presents an adaptive BOVW modelling, in which image feature extraction is achieved using deep feature learning and the amount of computation required for the development of visual codebook is minimised using a batch implementation of particle swarm optimisation. The proposed method is tested using Caltech-101 image dataset, and the results confirm the suitability of the proposed method in improving the categorisation performance while reducing the computational load.

A hybrid codebook model for object categorization using two-way clustering based codebook generation method

Both the visual codebook and the codebook model are considered as two main parts of most object classification frameworks. In the original codebook model, each image descriptor is encoded using a single codebook obtained usually using a clustering approach. In this paper, we propose a hybrid codebook model for an object classification task. A simultaneous clustering approach is applied to image descriptors to generate two variant codebooks and used them separately to encode and represent an image through a patch-based codebook model and a feature-based codebook model respectively. The proposed codebook model has been tested on the Caltech-101 dataset. Experimental results demonstrate state-of-the-art performance compared to typical clustering-based codebook model.

Efficient Technique for word identification and recognition in Telugu Documents

Telugu language is one of the most spoken Indian languages throughout the world. Since it has an old heritage, so Telugu literature and newspaper publications can be scanned to identify individual words. Identification of Telugu word images poses serious problems owing to its complex structure and larger set of individual characters. This paper aims to develop a novel methodology to achieve the same using SIFT (Scale Invariant Feature Transform) features of telugu words and classifying these features using BoVW (bag of visual words). The features are clustered to create a dictionary using k-means clustering. These words are used to create a visual codebook of the word images and the classification is achieved through SVM (Support Vector Machine).

Codebook-Free Compact Descriptor for Scalable Visual Search

The MPEG compact descriptors for visual search (CDVS) is a standard toward image matching and retrieval. To achieve high retrieval accuracy over a large scale image/video dataset, recent research efforts have demonstrated that employing extremely high-dimensional descriptors such as the Fisher vector (FV) and the vector of locally aggregated descriptors (VLAD) can yield good performance. Since the FV (or VLAD) possesses high discriminability but small visual vocabulary, it has been adopted by CDVS to construct a global compact descriptor. In this paper, we study the development of global compact descriptors in the completed CDVS standard and the emerging compact descriptors for video analysis (CDVA) standard, in which we formulate the FV (or VLAD) compression as a resource-constrained optimization problem. Accordingly, we propose a codebook-free aggregation method via dual selection to generate a global compact visual descriptor, which supports fast and accurate feature matching free of large visual codebooks, fulfilling the low memory requirement of mobile visual search at significantly reduced latency. Specifically, we investigate both sample-specific Gaussian component redundancy and bit dependency within a binary aggregated descriptor to produce compact binary codes. Our technique contributes to the scalable compressed Fisher vector (SCFV) adopted by the CDVS standard. Moreover, the SCFV descriptor is currently serving as the frame-level hand-crafted video feature, which inspires the inheritance of CDVS descriptors for the emerging CDVA standard. Furthermore, we investigate the positive complementary effect of our standard compliant compact descriptor and deep learning based features extracted from convolutional neural networks with significant mean average precision gains. Extensive evaluation over benchmark databases shows the significant merits of the codebook-free binary codes for scalable visual search.

Improved Hough transform by modeling context with conditional random fields for partially occluded pedestrian detection

Traditional Hough transform-based methods detect objects by casting votes to object centroids from object patches. It is difficult to disambiguate object patches from the background by a classifier without contextual information, as an image patch only carries partial information about the object. To leverage the contextual information among image patches, we capture the contextual relationships on image patches through a conditional random field (CRF) with latent variables denoted by locality-constrained linear coding (LLC). The strength of the pairwise energy in the CRF is measured using a Gaussian kernel. In the training stage, we modulate the visual codebook by learning the CRF model iteratively. In the test stage, the binary labels of image patches are jointly estimated by the CRF model. Image patches labeled as the object category cast weighted votes for object centroids in an image according to the LLC coefficients. Experimental results on the INRIA pedestrian, TUD Brussels, and Caltech pedestrian datasets demonstrate the effectiveness of the proposed method compared with other Hough transform-based methods.

Mobile Visual Search Based on Histogram Matching and Zone Weight Learning

In this paper, we propose a novel image retrieval algorithm for mobile visual search. At first, a short visual codebook is generated based on the descriptor database to represent the statistical information of the dataset. Then, an accurate local descriptor similarity score is computed by merging the tf-idf weighted histogram matching and the weighting strategy in compact descriptors for visual search (CDVS). At last, both the global descriptor matching score and the local descriptor similarity score are summed up to rerank the retrieval results according to the learned zone weights. The results show that the proposed approach outperforms the state-of-the-art image retrieval method in CDVS.

Learning visual codebooks for image classification using spectral clustering

This study explores the idea of learning visual codebook using spectral clustering, which we call spectral visual codebook learning (SVCL). Though spectral clustering has been widely applied into unsupervised segmentation, clustering, and manifold learning, using it to learn codebooks on standard image benchmark datasets has not been thoroughly studied. We show how learned codebooks by SVCL can be used for scene classification, texture recognition and image categorization. We describe several implementations for constructing the similarity graph and addressing the large-scale local image patches problem. We show that our approach captures nonlinear manifolds of semantic image patches. Another advantage is that both label and spatial information can be incorporated without increasing its model complexity. We validate SVCL on datasets such as KTH-TIPS, Scene-15, Graz-02, and Caltech-101.

Identity Verification Based on Facial Pose Pool and Bag of Words Model

There are many challenges for face based identity verification. It is one of fundamental topics in image processing and video analysis, and so on. A novel approach has been developed for facial identity verification based on a facial pose pool, which is constructed in an incremental clustering way to find both facial spatial information and orientation diversity. Bag of words is selected to extract image features from the facial pose pool in affine SIFT descriptor. The visual codebook is generated ink-means and Gaussian mixture model. Posterior pseudo probabilities are used to compute the similarities between each visual word and corresponding local features for image representation. Comparisons with some state-of-the-arts have highlighted the superior performance of the proposed method.

Detecting Damaged Building Regions Based on Semantic Scene Change from Multi-Temporal High-Resolution Remote Sensing Images

The detection of damaged building regions is crucial to emergency response actions and rescue work after a disaster. Change detection methods using multi-temporal remote sensing images are widely used for this purpose. Differing from traditional methods based on change detection for damaged building regions, semantic scene change can provide a new point of view since it can indicate the land-use variation at the semantic level. In this paper, a novel method is proposed for detecting damaged building regions based on semantic scene change in a visual Bag-of-Words model. Pre- and post-disaster scene change in building regions are represented by a uniform visual codebook frequency. The scene change of damaged and non-damaged building regions is discriminated using the Support Vector Machine (SVM) classifier. An evaluation of experimental results, for a selected study site of the Longtou hill town of Yunnan, China, which was heavily damaged in the Ludian earthquake of 14 March 2013, shows that this method is feasible and effective for detecting damaged building regions. For the experiments, WorldView-2 optical imagery and aerial imagery is used.

Multimodality semantic segmentation based on polarization and color images

Semantic segmentation gives a meaningful class label to every pixel in an image. It enables intelligent devices to understand the scene and has received sufficient attention during recent years. Traditional imaging systems always apply their methods on RGB, RGB-D or even RGB combined with geometric information. However, for outdoor applications, strong reflection or poor illumination appears to reduce the visualization of the real shape or texture of the objects, thus limiting the performance of semantic segmentation algorithms. To tackle this problem, this paper adopts polarization imaging as it can provide complementary information by describing some imperceptible light properties, which varies from different materials. For acceleration, SLIC superpixel segmentation is used to speed up the system. HOG and LBP features are extracted from both color and polarization images. After quantization using visual codebooks, Joint Boosting classifier is trained to label each pixel based on the quantized features. The proposed method was evaluated both on Day-set and Dusk-set. The experimental results show that using polarization setup can provide complementary information to improve the semantic segmentation accuracy. Especially, a large improvement on Dusk-set shows its capacity for intelligent vehicle applications under dark illumination condition.

Construction of visual codebook to speed up visual-based simultaneous localization and mapping

AbstractWe propose a visual-based self-localization system that requires less computational time and memory resources when utilized in an indoor environment. In this study, localization is accomplished by comparing an observation with a significant amount of preprocessed images. The larger the database, the slower the self-location calculation. We constructed an economical database (codebook for local image features), in terms of computational time and memory usage, to resolve this challenge. We sequentially registered new codes to avoid duplication. Additionally, we indexed the codebook to speed up the calculation when using an approximating search. The experiment result illustrated that the strategies of reducing the size of the codebook and approximating the calculation contributed to reducing the calculation cost and improving the self-localization accuracy.

Object classification using a local texture descriptor and a support vector machine

Objects classification or object detection is one of the most challenging tasks in computer vision. Digital images taken of real-life scenes capture objects at different positions, rotations and scales. Furthermore, variations in lighting, shape, color and texture within the same class make object classification a huge obstacle for computer vision algorithms. The most robust methodologies related to variations in lighting, rotation, color and scale, are based on “key points” localization, followed by applying a local descriptor to each surrounding region. Researchers have used various methods for detecting key points and have applied various local descriptors. Of these, the Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF) and Center-Symmetric Local Binary Pattern (CS-LBP) methods have obtained good performance and are associated with clustering algorithms or histogram representation based on independent features (Bag of Features (BoF)). In the BoF approach, the visual codebook extracted around the “key points” regions can effectively describe objects by their appearance based on local texture analysis. Recently, we proposed two new texture descriptors for object detection based on the Local Mapped Pattern (LMP) approach. The Mean-Local Mapped Pattern (M-LMP) and the Center Symmetric Local Mapped Pattern (CS-LMP) exhibit better performance than SIFT and CS-LBP, but prior results have shown that the size of descriptors could be reduced without loss of sensitivity. In this paper, we investigated reducing the size of the M-LMP descriptor and then evaluating its performance for object classification by a Support Vector Machine (SVM) classifier. In our experiments, we implemented an object recognition system based on the M-LMP reduced descriptor, and compared our results against the SIFT, Local Intensity Order Pattern (LIOP) and CS-LMP descriptors. The object classification results were analyzed using a BoF model and a SVM classifier, with the result that performance using the reduced descriptor is better than the other three well-known methods tested and also requires less processing time.

Improving content-based image retrieval with compact global and local multi-features

The accuracy of content-based image retrieval (CBIR) systems is significantly affected by the discriminatory power of image features and distance measures. This paper performs an investigation towards finding the best local and global features and distance measures for content-based image retrieval. It provides insights into the trade-offs regarding computational costs, memory utilization and accuracy on several standard datasets which include MIRFLICKR, Corel, Holidays and ZuBuD. First, low-dimensional global and local features are extracted individually to generate a bank of small image features. Second, multilevel descriptor forms are utilized to produce highly discriminative image representations based on multi-features aggregation scheme. The relationship is highlighted between features (local and global) and other retrieval factors such as quantization approaches, visual codebooks, distance measures, vectorization methods, memory and retrieval speed. The resulting composite image representations are compact, i.e., only 32–64 vector dimension and 32–128 codebook size, and preserve high discriminative levels which further boost the retrieval accuracy and performance. The experimental results show that the presented multi-features image representations are efficient and outperform many competitive methods of the state-of-the-art.

Visual Semantic Based 3D Video Retrieval System Using HDFS

This paper brings out a neoteric frame of reference for visual semantic based 3d video search and retrieval applications. Newfangled 3D retrieval application spotlight on shape analysis like object matching, classification and retrieval not only sticking up entirely with video retrieval. In this ambit, we delve into 3D-CBVR (Content Based Video Retrieval) concept for the first time. For this purpose, we intent to hitch on BOVW and Mapreduce in 3D framework. Instead of conventional shape based local descriptors, we tried to coalesce shape, color and texture for feature extraction. For this purpose, we have used combination of geometric & topological features for shape and 3D co-occurrence matrix for color and texture. After thriving extraction of local descriptors, TB-PCT (Threshold Based- Predictive Clustering Tree) algorithm is used to generate visual codebook and histogram is produced. Further, matching is performed using soft weighting scheme with L2 distance function. As a final step, retrieved results are ranked according to the Index value and acknowledged to the user as a feedback .In order to handle prodigious amount of data and Efficacious retrieval, we have incorporated HDFS in our Intellection. Using 3D video dataset, we future the performance of our proposed system which can pan out that the proposed work gives meticulous result and also reduce the time intricacy.

Scene classification using local and global features with collaborative representation fusion

This paper presents an effective scene classification approach based on collaborative representation fusion of local and global spatial features. First, a visual word codebook is constructed by partitioning an image into dense regions, followed by the typical k-means clustering. A locality-constrained linear coding is employed on dense regions via the visual codebook, and a spatial pyramid matching strategy is then used to combine local features of the entire image. For global feature extraction, the method called multiscale completed local binary patterns (MS-CLBP) is applied to both the original gray scale image and its Gabor feature images. Finally, kernel collaborative representation-based classification (KCRC) is employed on the extracted local and global features, and class label of the testing image is assigned according to the minimal approximation residual after fusion. The proposed method is evaluated by using four commonly-used datasets including two remote sensing images datasets, an indoor and outdoor scenes dataset, and a sports action dataset. Experimental results demonstrate that the proposed method significantly outperforms the state-of-the-art methods.