Bag Of Visual Words Research Articles

Content-Based Image Retrieval System using Color Moment and Bag of Visual Words with Local Binary Pattern

Researchers have recently focused their attention on Content-Based Image Retrieval (CBIR). It has emerged as one of the most fascinating areas in image processing and computer vision. With CBIR, the most comparable pictures that match the query image are pulled from an image database. As a result, it necessitates feature extraction (Local / Global) and similarity calculation. This paper uses a CBIR technique to determine the images that best match the image query by utilizing both global and local image features. A color moment is used for global features to describe the complete image. Local Binary Pattern (LBP) as a local feature, on the other hand, extracts interest points by building a Bag of Visual Words (BoVW). The distance between the query and database image features is computed using the Euclidean distance. Precision and recall are computed on the Corel-1K dataset to assess the retrieval performance.

A new method of construction waste classification based on two-level fusion.

The automatic sorting of construction waste (CW) is an essential procedure in the field of CW recycling due to its remarkable efficiency and safety. The classification of CW is the primary task that guides automatic and precise sorting. In our work, a new method of CW classification based on two-level fusion is proposed to promote classification performance. First, statistical histograms are used to obtain global hue information and local oriented gradients, which are called the hue histogram (HH) and histogram of oriented gradients (HOG), respectively. To fuse these visual features, a bag-of-visual-words (BoVW) method is applied to code HOG descriptors in a CW image as a vector, and this process is named B-HOG. Then, based on feature-level fusion, we define a new feature to combine HH and B-HOG, which represent the global and local visual characteristics of an object in a CW image. Furthermore, two base classifiers are used to learn the information from the color feature space and the new feature space. Based on decision-level fusion, we propose a joint decision-making model to combine the decisions from the two base classifiers for the final classification result. Finally, to verify the performance of the proposed method, we collect five types of CW images as the experimental data set and use these images to conduct experiments on three different base classifiers. Moreover, we compare this method with other extant methods. The results demonstrate that our method is effective and feasible.

Deep Learning-Based BoVW–CRNN Model for Lung Tumor Detection in Nano-Segmented CT Images

One of the most common oncologies analyzed among people worldwide is lung malignancy. Early detection of lung malignancy helps find a suitable treatment for saving human lives. Due to its high resolution, greater transparency, and low noise and distortions, Computed Tomography (CT) images are most commonly used for processing. In this context, this research work mainly focused on the multifaceted nature of lung cancer diagnosis, a quintessential, fascinating, and risky subject of oncology. The input used here has been nano-image, enhanced with a Gabor filter and modified color-based histogram equalization. Then, the image of lung cancer was segmented by using the Guaranteed Convergence Particle Swarm Optimization (GCPSO) algorithm. A graphical user interface nano-measuring tool was designed to classify the tumor region. The Bag of Visual Words (BoVW) and a Convolutional Recurrent Neural Network (CRNN) were employed for image classification and feature extraction processes. In terms of findings, we achieved the average precision of 96.5%, accuracy of 99.35%, sensitivity of 97%, specificity of 99% and F1 score of 95.5%. With the proposed solution, the overall time required for the segmentation of images was much smaller than the existing solutions. It is also remarkable that biocompatible-based nanotechnology was developed to distinguish the malignancy region on a nanometer scale and has to be evaluated automatically. That novel method succeeds in producing a proficient, robust, and precise segmentation of lesions in nano-CT images.

Multi-Subject Image Retrieval by Fusing Object and Scene-Level Feature Embeddings

Most existing image retrieval methods separately retrieve single images, such as a scene, content, or object, from a single database. However, for general purposes, target databases for image retrieval can include multiple subjects because it is not easy to predict which subject is entered. In this paper, we propose that image retrieval can be performed in practical applications by combining multiple databases. To deal with multi-subject image retrieval (MSIR), image embedding is generated through the fusion of scene- and object-level features, which are based on Detection Transformer (DETR) and a random patch generator with a deep-learning network, respectively. To utilize these feature vectors for image retrieval, two bags-of-visual-words (BoVWs) were used as feature embeddings because they are simply integrated with preservation of the characteristics of both features. A fusion strategy between the two BoVWs was proposed in three stages. Experiments were conducted to compare the proposed method with previous methods on conventional single-subject datasets and multi-subject datasets. The results validated that the proposed fused feature embeddings are effective for MSIR.

Earthquake-triggered landslide interpretation model of high resolution remote sensing imageries based on bag of visual word

Traditional visual interpretation is often inefficient due to its excessively workload professional knowledge and strong subjectivity. Therefore, building an automatic interpretation model on high spatial resolution remote sensing images is the key to the quick and efficient interpretation of earthquake-triggered landslides. Aiming at addressing this problem, a landslide interpretation model of high-resolution images based on bag of visual word (BoVW) feature was proposed. The high-resolution images were pre-processed, and then BoVW feature and support vector machine (SVM) was adopted to establish an automatic landslide interpretation model. This model was further compared with the currently widely used Histogram of Oriented Gradient(HoG) feature extraction model. In order to test the effectiveness of the method, typical landslide images were selected to construct a landslide sample library, which was subsequently utilized as the foundation for conducting an experimental study. The results show that the accuracy of landslide extraction using this method reaches as high as 89%, indicating that the method can be used for the automatic interpretation of landslides in disaster-prone areas, and has high practical value for regional disaster prevention and damage reduction.

Determination of Moisture in Rice Grains Based on Visible Spectrum Analysis

Rice grain production is important for the world economy. Determining the moisture content of the grains, at several stages of production, is crucial for controlling the quality, safety, and storage of the grain. This work inspects how well rice images from global and local descriptors work for determining the moisture content of the grains using artificial vision and intelligence techniques. Three sets of images of rice grains from the INIAP 12 variety (National Institute of Agricultural Research of Ecuador) were captured with a mobile camera. The first one with natural light and the other ones with a truncated pyramid-shaped structure. Then, a set of global descriptors (color, texture) and a set of local descriptors (AZAKE, BRISK, ORB, and SIFT) in conjunction with the dominate technique bag of visual words (BoVW) were used to analyze the content of the image with classification and regression algorithms. The results show that detecting humidity through images with classification and regression algorithms is possible. Finally, f1-score values of at least 0.9 were accomplished for global color descriptors and of 0.8 for texture descriptors, in contrast to the local descriptors (AKAZE, BRISK, and SIFT) that reached up to an f1-score of 0.96.

In-Motion Railroad Tie Deflection Measurement via Ultrasonic Airborne Sonar and Computer Vision Techniques

ABSTRACT It is known in the railroad maintenance engineering community that the deflection of railroad ties is an indicator of the quality of the tie–ballast interface, whose deterioration may cause dangerous train derailments. A new technology is proposed to reconstruct the full-field deflection profile of railroad ties in-motion by means of non-contact ultrasonic testing and computer vision techniques. The sensing layout consists of an array of air-coupled capacitive transducers (operated in pulse-echo sonar-based ranging mode) and a high frame-rate camera, rigidly connected to the main frame of a moving train car. The acquisition system is programmed such that the synchronized waveforms and images are collected and saved as train car moves. In the processing stage, a supervised machine learning-based image classification approach is developed to demarcate the tie boundaries. For this purpose, the Speeded-Up Robust Features (SURF) and Bag of Visual Words (BOVW) algorithms are employed to encode images into condensed feature vectors, which are subsequently fed into the Support Vector Machine (SVM) to train a classifier. The relative deflections of the identified ties are eventually computed by tracking the time-of-flight of the reflected waves from the surfaces flagged as tie. An image processing technique is also developed to estimate the spatial resolution of the tracking system, required to reconstruct the full-field deflection profile of the scanned ties. The importance of such a technique is stressed if the test run is performed without any dedicated positioning system. The proposed ‘tie sonar’ system was prototyped and used to reconstruct the deflection profile of the ties scanned during a series of test runs conducted at slow (walking) speed at the Rail Defect Testing Facility (RDTF) of UC San Diego as well as a BNSF yard in San Diego, CA, with a realistic train load. Further developments of this system should include a performance evaluation at higher speeds (e.g., revenue speed).

Multi-Model Fusion of Encoding Methods-Based Visual Words for Target Recognition in Infrared Images

The limited information contained in infrared images present a serious problem, therefore it is necessary to form a powerful feature descriptor that allows extracting the maximum information and describing the image efficiently. To address this challenge, we propose a novel approach named multi-model fusion of encoding methods (MMFEM). First, several encoding methods for Bag Of Visual Words (BOVW) model were evaluated. Then, we fuse the best encoding methods obtained using three levels of fusion: feature-level fusion, decision-level fusion and hybrid-level fusion. Finally, the outputs of the fusion process were used to form a final decision for target recognition in infrared images. Two infrared datasets were employed to evaluate the performance of the proposed approach. The first one is Visible and Infrared Spectrum (VAIS) dataset comprising six categories of ships and the second dataset is a subset of Forward-Looking InfraRed (FLIR) thermal dataset comprising two object categories, vehicles and pedestrians. The proposed approach has exceed the state of the art for both datasets and we have reached 96.96% for FLIR and 71.26% for VAIS in overall classification accuracy.

Multi-Scale Feature Fusion for Interior Style Detection

Text-based search engines can extract various types of information when a user enters an appropriate search query. However, a text-based search often fails in image retrieval when image understanding is needed. Deep learning (DL) is often used for image task problems, and various DL methods have successfully extracted visual features. However, as human perception differs for each individual, a dataset with an abundant number of images evaluated by human subjects is not available in many cases, although DL requires a considerable amount of data to estimate space ambiance, and the DL models that have been created are difficult to understand. In addition, it has been reported that texture is deeply related to space ambiance. Therefore, in this study, bag of visual words (BoVW) is used. By applying a hierarchical representation to BoVW, we propose a new interior style detection method using multi-scale features and boosting. The multi-scale features are created by combining global features from BoVW and local features that use object detection. Experiments on an image understanding task were conducted on a dataset consisting of room images with multiple styles. The results show that the proposed method improves the accuracy by 0.128 compared with the conventional method and by 0.021 compared with a residual network. Therefore, the proposed method can better detect interior style using multi-scale features.

Keypoints class distribution based entropy for weighting scheme on image classification

In bag-of-visual words (BoVW), a weighting scheme is applied to improve the discriminative power of visual words, which affect the performance of image classification. However, the weighting schemes in the BoVW model did not utilize the information class of keypoints. In this paper, we propose an algorithm to measure the certainty of visual words which employ the information class of keypoints, called keypoints class distribution based entropy (KCDE). The algorithm is then combined with the existing weighting scheme namely term frequency-term distribution (TF-TD). The results show that the proposed weighting schemes give better classification results than the baseline weighting schemes.

Vibration-based and computer vision-aided nondestructive health condition evaluation of rail track structures

In railway engineering, monitoring the health condition of rail track structures is crucial to prevent abnormal vibration issues of the wheel–rail system. To address the problem of low efficiency of traditional nondestructive testing methods, this work investigates the feasibility of the computer vision-aided health condition monitoring approach for track structures based on vibration signals. The proposed method eliminates the tedious and complicated data pre-processing including signal mapping and noise reduction, which can achieve robust signal description using numerous redundant features. First, the method converts the raw wheel–rail vibration signals directly into two-dimensional grayscale images, followed by image feature extraction using the FAST-Unoriented-SIFT algorithm. Subsequently, Visual Bag-of-Words (VBoW) model is established based on the image features, where the optimal parameter selection analysis is implemented based on fourfold cross-validation by considering both recognition accuracy and stability. Finally, the Euclidean distance between word frequency vectors of testing set and the codebook vectors of training set is compared to recognize the health condition of track structures. For the three health conditions of track structures analyzed in this paper, the overall recognition rate could reach 96.7%. The results demonstrate that the proposed method performs higher recognition accuracy and lower bias with strong time-varying and random vibration signals, which has promising application prospect in early-stage structural defect detection.

An IoT based authentication system for therapeutic herbs measured by local descriptors using machine learning approach

The work aims to develop an automatic recognition model to classify medicinal plants using machine learning techniques to enrich the traditional medical system of India. Though many countries have accepted conventional medicine as the best alternative to synthetic drugs, there exists limitations such as lack of awareness among general public and unavailability of easy access to its source evidences that has led to its limited acceptance and usability. Herein, an intelligent system is proposed to use Raspberry Pi 3 Model B+ (RPi) and the RPi camera to capture the leaf images of Indian medicinal herbs and reveal their medical properties. Five types of models implemented to identify the medicinal plants. One of the models proposed as Herbmodel extracts a feature map from a captured medicinal leaf by combining three different feature extraction techniques, namely, Scale Invariant Feature Transform (SIFT), Oriented FAST and Rotated BRIEF (ORB) and histogram of oriented gradients on support vector machine as a classifier, predicts an average accuracy of 96.22% over a custom medicinal leaf dataset of 40 different species containing 2515 samples. Generate Bag of Visual Words (BoVW) by applying K-Means clustering on both SIFT and ORB descriptors to reduce the dimensionality. The combined feature vector is further analysed using random forest and k-nearest neighbor classifier. The efficacy of the proposed approach is benchmarked using Flavia dataset and artificial neural network (ANN) as a classifier. Our findings prove that the combination of local descriptors is an efficient measurement approach that benefits automatic recognition of plants based on leaf images. Also, a reliable source of medicinal leaf datasets with good quality leaf images is necessary to establish a machine learning model for medicinal plants.

Identification of malware families using stacking of textural features and machine learning

The growing rate of malware and its complexity demands a new approach to detecting evolving malware instead of relying only on high-level features such as opcodes, API calls, control flow graphs, etc. Moreover, extracting these features is an expensive and time-consuming because it requires disassembling or code execution. This paper presents a new malware detection architecture using image analysis & machine learning, requiring significantly fewer resources and does not depend upon code disassembling or execution. The collected binary samples are systematically labeled using the AVClass tool and clustering method based on similar characteristics of them. The labeled malicious program is visualized into grayscale images to extract the local and global textural features. The local textural features are extracted using SIFT, KAZE, and ORB descriptors, and global features are extracted using GIST, Hu Moments, and HOG. A bag of visual words (BoVW) algorithm is designed to select low-dimensional features and construct a local feature map of malware grayscale images. The feature maps from different image descriptors are stacked and used to train five machine learning algorithms, namely, k-Nearest Neighbor (k-NN), Support Vector Machine(SVM), Random Forest (RF), Naive Bayes(NB), and ExtraTree classifier. Two datasets are used for evaluations — the public MalImg dataset of 9339 samples of 25 different families and 690 real-world malware of 22 families collected on honeypots in the wild. Intensive experiments are performed for image descriptors, image ratios, vocabulary size, and computational time as a mean time to detection(MTTD) to devise the best detector. The proposed method obtained test accuracy of 98.34% with stacked global features and 98.23% with stacked local features. Test accuracy of 92.75% with low false-positive rates is obtained for real-world recent malware datasets. Experiment results reveal the efficacy of the proposed method in detecting polymorphic obfuscated malware. Finally, a comparison with other similar malware detection systems is presented.

Bubble Plume Target Detection Method of Multibeam Water Column Images Based on Bags of Visual Word Features

Bubble plumes, as main manifestations of seabed gas leakage, play an important role in the exploration of natural gas hydrate and other resources. Multibeam water column images have been widely used in detecting bubble plume targets in recent years because they can wholly record water column and seabed backscatter strengths. However, strong noises in multibeam water column images cause many issues in target detection, and traditional target detection methods are mainly used in optical images and are less efficient for noise-affected sonar images. To improve the detection accuracy of bubble plume targets in water column images, this study proposes a target detection method based on the bag of visual words (BOVW) features and support vector machine (SVM) classifier. First, the characteristics of bubble plume targets in water column images are analyzed, with the conclusion that the BOVW features can well express the gray scale, texture, and shape characteristics of bubble plumes. Second, the BOVW features are constructed following steps of point description extraction, description clustering, and feature encoding. Third, the quadratic SVM classifier is used for the recognition of target images. Finally, a procedure of bubble plume target detection in water column images is described. In the experiment using the measured data in the Strait of Georgia, the proposed method achieved 98.6% recognition accuracy of bubble plume targets in validation sets, and 91.7% correct detection rate of the targets in water column images. By comparison with other methods, the experimental results prove the validity and accuracy of the proposed method, and show potential applications of our method in the exploration and research on ocean resources.

Performance Analysis of Supervised Machine Learning Algorithms for Automatized Radiographical Classification of Maxillary Third Molar Impaction

Background: Oro-antral communication (OAC) is a common complication following the extraction of upper molar teeth. The Archer and the Root Sinus (RS) systems can be used to classify impacted teeth in panoramic radiographs. The Archer classes B-D and the Root Sinus classes III, IV have been associated with an increased risk of OAC following tooth extraction in the upper molar region. In our previous study, we found that panoramic radiographs are not reliable for predicting OAC. This study aimed to (1) determine the feasibility of automating the classification (Archer/RS classes) of impacted teeth from panoramic radiographs, (2) determine the distribution of OAC stratified by classification system classes for the purposes of decision tree construction, and (3) determine the feasibility of automating the prediction of OAC utilizing the mentioned classification systems. Methods: We utilized multiple supervised pre-trained machine learning models (VGG16, ResNet50, Inceptionv3, EfficientNet, MobileNetV2), one custom-made convolutional neural network (CNN) model, and a Bag of Visual Words (BoVW) technique to evaluate the performance to predict the clinical classification systems RS and Archer from panoramic radiographs (Aim 1). We then used Chi-square Automatic Interaction Detectors (CHAID) to determine the distribution of OAC stratified by the Archer/RS classes to introduce a decision tree for simple use in clinics (Aim 2). Lastly, we tested the ability of a multilayer perceptron artificial neural network (MLP) and a radial basis function neural network (RBNN) to predict OAC based on the high-risk classes RS III, IV, and Archer B-D (Aim 3). Results: We achieved accuracies of up to 0.771 for EfficientNet and MobileNetV2 when examining the Archer classification. For the AUC, we obtained values of up to 0.902 for our custom-made CNN. In comparison, the detection of the RS classification achieved accuracies of up to 0.792 for the BoVW and an AUC of up to 0.716 for our custom-made CNN. Overall, the Archer classification was detected more reliably than the RS classification when considering all algorithms. CHAID predicted 77.4% correctness for the Archer classification and 81.4% for the RS classification. MLP (AUC: 0.590) and RBNN (AUC: 0.590) for the Archer classification as well as MLP 0.638) and RBNN (0.630) for the RS classification did not show sufficient predictive capability for OAC. Conclusions: The results reveal that impacted teeth can be classified using panoramic radiographs (best AUC: 0.902), and the classification systems can be stratified according to their relationship to OAC (81.4% correct for RS classification). However, the Archer and RS classes did not achieve satisfactory AUCs for predicting OAC (best AUC: 0.638). Additional research is needed to validate the results externally and to develop a reliable risk stratification tool based on the present findings.

Indian Sign Language recognition system using SURF with SVM and CNN

Hand signs are an effective form of human-to-human communication that has a number of possible applications. Being a natural means of interaction, they are commonly used for communication purposes by speech impaired people worldwide. In fact, about one percent of the Indian population belongs to this category. This is the key reason why it would have a huge beneficial effect on these individuals to incorporate a framework that would understand Indian Sign Language. In this paper, we present a technique that uses the Bag of Visual Words model (BOVW) to recognize Indian sign language alphabets (A-Z) and digits (0–9) in a live video stream and output the predicted labels in the form of text as well as speech. Segmentation is done based on skin colour as well as background subtraction. SURF (Speeded Up Robust Features) features have been extracted from the images and histograms are generated to map the signs with corresponding labels. The Support Vector Machine (SVM) and Convolutional Neural Networks (CNN) are used for classification. An interactive Graphical User Interface (GUI) is also developed for easy access.

Location recognition in laparoscopic surgery

Abstract Navigation systems play an increasingly important role in minimally invasive surgery (MIS) by mitigating the problems rising from the decoupling of hand-eye movement of the surgeon. Many of these systems suffer from a high dependency on external optical tracking systems that require a constant line-of-sight to the optical markers being tracked. Simultaneous localization and mapping (SLAM) algorithms allow tracking the endoscope in cases where optical tracking fails due to the cluttered environment in the operating room. To ensure a correct camera pose estimate and to correct for drift, a recognition of previously visited locations (loop closures) is essential. We propose a method for location recognition in a minimally invasive scenario that only requires a stereo endoscope and an inertial measurement unit (IMU).We use a hierarchical bag-of-visual-words (BoW) algorithm that saves compact image representations and enables querying for matching images. A two-staged consistency check using a random sample consensus (RANSAC) and the data measured by the IMU ensure a high matching precision.

Novel content based medical image retrieval based on BoVW classification method

Amount of digital image is increasing exponentially nowadays. Storage requirements for these images increase from Gigabytes to Pet bytes. Searching and retrieving the relevant images from such larger volume of images datasets based on its contents plays dynamic role in various applications for computer vision. There are many common retrieval systems are there to provide solutions for this problem. Times taken to retrieve the images are high and accuracy of retrieved images is less in the existing systems. The limitation of Hadoop maps reduce is the lack of performing real time tasks efficiently. A proficient content-based image retrieval framework based on Spark Map Reduce with bag of visual word is to be proposed to perform with high accuracy for big data. The Apache spark programming can be utilized to productively recover pictures with less retrieval time and retrieve the accurate images from the big database that resemble the query image. The proposed system obtained an accuracy of 97.7% and it is better than the existing classification method for Retrieve the image from the large database.

Intelligence Classification Algorithm-Based Drug-Resistant Pulmonary Tuberculosis Computed Tomography Imaging Features and Influencing Factors.

The drug resistance and influencing factors of patients with pulmonary tuberculosis were investigated, and a dual attention dilated residual network (DADRN) algorithm was proposed. The algorithm was applied to process and analyze lung computed tomography (CT) images of 400 included patients with pulmonary tuberculosis. Besides, sparse code book algorithm and bag of visual word (BOVW) algorithms were introduced and compared, and the influencing factors of pulmonary tuberculosis drug resistance were analyzed. The results demonstrated that the localization precision of lung consolidation, nodules, and cavities by the DADRN algorithm reached 91.2%, 92.5%, and 93.8%, respectively. The recall rate of the three algorithms amounted to 83.55%, 84.5%, and 86.4%, respectively. Both localization precision and recall rate of the DADRN algorithm were higher than those of other two algorithms (P < 0.05). The drug resistance rate of streptomycin, isoniazid, and rifampin of the patients aged between 40 and 59 was all higher than those of the patients in other age groups. The drug resistance rate of streptomycin, isoniazid, and rifampin of retreated patients was all higher than those of patients initially treated. The drug resistance rate of streptomycin, isoniazid, and rifampin of the patients with tuberculosis contact was all higher than those of the patients without tuberculosis contact (P < 0.05). Based on the above results, the accuracy of CT images processed by dual attention-based dilated residual classification network algorithm was higher than that processed by other two algorithms. Age, medical history, and history of exposure to tuberculosis were the influencing factors of the drug resistance of patients with pulmonary tuberculosis.

Encoding hieroglyph segments to represent hieroglyphs following the bag of visual word model for retrieval

The representation of hieroglyphs in retrieval systems represents a great challenge since these systems’ results highly depend on the used representation. In the literature, the most successful works in this area compute local descriptors from a set of points of interest taken from the entire hieroglyph and use them under the BoVW model to represent hieroglyphs. Unlike these works, this paper presents a way to extract segments from hieroglyphs and, by encoding the extracted segments through local descriptors, proposes to use them under the Bag of Visual Words (BoVW) model to represent hieroglyphs. Our experiments show that the proposed representation allows us to obtain retrieval results that overcome those reported by using state of the art representations.