Query By Example Research Articles

Hough Transform-Based Angular Features for Learning-Free Handwritten Keyword Spotting.

Handwritten keyword spotting (KWS) is of great interest to the document image research community. In this work, we propose a learning-free keyword spotting method following query by example (QBE) setting for handwritten documents. It consists of four key processes: pre-processing, vertical zone division, feature extraction, and feature matching. The pre-processing step deals with the noise found in the word images, and the skewness of the handwritings caused by the varied writing styles of the individuals. Next, the vertical zone division splits the word image into several zones. The number of vertical zones is guided by the number of letters in the query word image. To obtain this information (i.e., number of letters in a query word image) during experimentation, we use the text encoding of the query word image. The user provides the information to the system. The feature extraction process involves the use of the Hough transform. The last step is feature matching, which first compares the features extracted from the word images and then generates a similarity score. The performance of this algorithm has been tested on three publicly available datasets: IAM, QUWI, and ICDAR KWS 2015. It is noticed that the proposed method outperforms state-of-the-art learning-free KWS methods considered here for comparison while evaluated on the present datasets. We also evaluate the performance of the present KWS model using state-of-the-art deep features and it is found that the features used in the present work perform better than the deep features extracted using InceptionV3, VGG19, and DenseNet121 models.

Handwritten keyword spotting using deep neural networks and certainty prediction

Word spotting using deep Convolutional Neural Networks (CNN) has recently obtained significant results in handwritten documents retrieval application. In this paper, we propose a novel word spotting method based on Monte-Carlo dropout CNN to compute the certainty of extracted features that can be used in both query-by-example (QBE) and query-by-string (QBS) word spotting scenarios. In the QBE and during the training, an adaptable certainty threshold is assigned for the words of each class. Cosine distance between the predicted certainty of the query image and the retrieval set is compared with the certainty threshold of each class in the matching step. For the QBS, the query class is compared to the class of the retrieval set obtained by the certainty prediction. We evaluated our proposed method on four public handwritten databases. Experimental results showed that the accuracy achieved in both QBE and QBS scenarios outperforms the state-of-the-art methods.

Fast Query-by-example Speech Search using Attention-based Deep Binary Embeddings

State-of-the-art query-by-example (QbE) speech search approaches usually use recurrent neural network (RNN) based acoustic word embeddings (AWEs) to represent variable-length speech segments with fixed-dimensional vectors, and thus simple cosine distances can be measured over the embedded vectors of both the spoken query and the search content. In this paper, we aim to improve search accuracy and speed for the AWE-based QbE approach in low-resource scenario. First, multi-head self-attentive mechanism is introduced for learning a sequence of attention weights for all time steps of RNN outputs while attending to different positions of a speech segment. Second, as the real-valued AWEs suffer from substantial computation in similarity measure, a hashing layer is adopted for learning deep binary embeddings, and thus binary pattern matching can be directly used for fast QbE speech search. The proposed approach of self-attentive deep hashing network is effectively trained with three specifically-designed objectives: a penalization term, a triplet loss, and a quantization loss. Experiments show that our approach improves the relative search speed by 8 times and mean average precision (MAP) by 18.9%, as compared with the previous best real-valued embedding approach.

Example-driven query intent discovery

Traditional relational data interfaces require precise structured queries over potentially complex schemas. These rigid data retrieval mechanisms pose hurdles for non-expert users, who typically lack language expertise and are unfamiliar with the details of the schema. Query by Example (QBE) methods offer an alternative mechanism: users provide examples of their intended query output and the QBE system needs to infer the intended query. However, these approaches focus on the structural similarity of the examples and ignore the richer context present in the data. As a result, they typically produce queries that are too general, and fail to capture the user's intent effectively. In this paper, we present SQuID, a system that performs semantic similarity-aware query intent discovery . Our work makes the following contributions: (1) We design an end-to-end system that automatically formulates select-project-join queries in an open-world setting, with optional group-by aggregation and intersection operators; a much larger class than prior QBE techniques. (2) We express the problem of query intent discovery using a probabilistic abduction model , that infers a query as the most likely explanation of the provided examples. (3) We introduce the notion of an abduction-ready database, which precomputes semantic properties and related statistics, allowing SQuID to achieve real-time performance. (4) We present an extensive empirical evaluation on three real-world datasets, including user-intent case studies, demonstrating that SQuID is efficient and effective, and outperforms machine learning methods, as well as the state-of-the-art in the related query reverse engineering problem.

Word spotting and recognition via a joint deep embedding of image and text

This work addresses three important yet challenging problems of handwritten text understanding: word recognition, query-by-example (QBE) word spotting and query-by-string (QBS) word spotting. In most existing approaches, these related tasks are considered independently. We propose a single unified framework based on deep learning to solve all three tasks efficiently and simultaneously. In this framework, an end-to-end deep neural network architecture is used for the joint embedding of handwritten word texts and images. Word images are embedded via a convolution neural network (CNN), which is trained to predict a representation modeling character-level information. The output of the last convolutional layer is considered as representation in the joint embedding subspace. Likewise, a recurrent neural network (RNN) is used to map a sequence of characters to the joint subspace representation. Finally, a model based on multi-layer perceptrons is proposed to predict the matching probability between two embedding vectors. Experiments on five databases of documents written in three languages show our method to yield state-of-the-art performance for QBE and QBS word spotting. The proposed method also obtains competitive results for word recognition, when compared against approaches tailored specifically for this task.

Subgraph spotting in graph representations of comic book images

Graph-based representations are the most powerful data structures for extracting, representing and preserving the structural information of underlying data. Subgraph spotting is an interesting research problem, especially for studying and investigating the structural information based content-based image retrieval (CBIR) and query by example (QBE) in image databases. In this paper we address the problem of lack of freely available ground-truthed datasets for subgraph spotting and present a new dataset for subgraph spotting in graph representations of comic book images (SSGCI) with its ground-truth and evaluation protocol. Experimental results of two state-of-the-art methods of subgraph spotting are presented on the new SSGCI dataset.

A Query Language for Data Access in Ubiquitous Environments


 
 
 Recent progress in ubiquitous computing technology is allowing development of new and so- phisticated applications that are now able to store very large data on the mobile device itself, besides access data stored remotely. Thus emerges the need to query and maintain data from portable equipments. However, traditional query languages such as SQL (Structure Query Lan- guage) and QbE (Query by Example) are inadequate for devices with small screens. Hence, this work presents QbZ (Query by Zoom), a query language based on Semantic Zoom for small-screen devices. The proposed mechanism is particularly suitable to locate and access data stored in portable equipments. In order to demonstrate the benefits of using QbZ, usability tests were conducted, as well as a comparative analysis with the existing solutions.
 
 

Efficient Learning-Free Keyword Spotting.

In this article, a method for segmentation-based learning-free Query by Example (QbE) keyword spotting on handwritten documents is proposed. The method consists of three steps, namely preprocessing, feature extraction and matching, which address critical variations of text images (e.g., skew, translation, different writing styles). During the feature extraction step, a sequence of descriptors is generated using a combination of a zoning scheme and a novel appearance descriptor, referred as modified Projections of Oriented Gradients. The preprocessing step, which includes contrast normalization and main-zone detection, aims to overcome the shortcomings of the appearance descriptor. Moreover, an uneven zoning scheme is introduced by applying a denser zoning only on query images for a more detailed representation. This leads to a significant reduction in storage requirements of a document collection. The distance between the query and word sequences is efficiently computed by the proposed Selective Matching algorithm. This algorithm is further extended to handle an augmented set of images originating from a single query image. The efficiency of the proposed method is demonstrated by experimentation conducted on seven publicly available datasets. In these experiments, the proposed method significantly outperforms all state-of-the-art learning-free techniques.

AN OVERVIEW OF AUDIO CLASSIFIER METHODS AND TECHNIQUES

A brief overview of trends and developments in the area of Content-Based Audio Indexing and Retrieval (CBAIR), during the past few years. Here we explored some limitations and constrains of existing Query by Example (QBE) and Query by Humming (QBH) CBAIR systems. We talked about different methods to represent musical objects, like feature-based representation, musical parameter-based representation; similarly retrieval strategies, like feature based retrieval as well as melody or theme based retrieval of musical objects, in this paper. Moreover, some important issues regarding to indexing and retrieval performance i.e. efficient indexing and retrieval complexity, in this area are discussed thoroughly. Finally, hypothetical solutions for the existing problems in this area are proposed to improve the performance.

Weighted fast sequential DTW for multilingual audio Query-by-Example retrieval

This paper examines multilingual audio Query-by-Example (QbE) retrieval, utilizing the posteriorgram-based Phonetic Unit Modelling (PUM) approach and the Weighted Fast Sequential Dynamic Time Warping (WFSDTW) algorithm. The PUM approach employs phone recognizers trained on language-specific external resources in a supervised way. Thus, the information about the phonetic distribution is embedded in the process of acoustic modelling. The resulting acoustic models were also used for language-independent QbE retrieval. The improved WFSDTW algorithm was implemented in order to perform retrievals for each query (keyword) within the particular utterance file. The major interest is placed on a retrieval performance measurement of the proposed WFSDTW solution employing posteriorgram-based keyword matching with Gaussian mixture modelling (GMM). Score normalization and fusion of four different language-dependent sub-systems was carried out using a simple max-score merging strategy. The results show a certain predominance of the proposed WFSDTW solution among two other evaluated techniques, namely basic DTW and segmental DTW algorithms. Also, the combination of multiple PUM techniques together with the WFSDTW has been proved as an effective solution for the QbE task.

DeepSketch 3

Freehand sketches are a simple and powerful tool for communication. They are easily recognized across cultures and suitable for various applications. In this paper, we use deep convolutional neural networks (ConvNets), state-of-the-art in the field of sketch recognition, to address several applications of automatic sketch processing: complete and partial sketch recognition, sketch retrieval using query-by-example (QbE), and sketch-based image retrieval (SBIR) i.e the retrieval of images using a QbE paradigm but where the query is a sketch. We first focus on improving sketch recognition. For this purpose we compare different ConvNet architectures, training paradigms and data fusion schemes. This enabled us to outperform previous state-of-the-art in two large scale benchmarks for sketch classification. We achieved a mean average accuracy of 79.18% for the TU-Berlin sketch benchmark and 93.02% for the sketchy database. For partial sketch recognition, we were able to produce a system that achieves a mean average accuracy of 52.58% with only 40% of the strokes. We then conduct a comprehensive study of ConvNets features to enhance sketch retrieval and image retrieval, using a kNN similarity search paradigm in the ConvNet feature space. For the sketch retrieval tasks, we compare the performance obtained with features extracted from various depths (ConvNet layers) using one of the best performing model from the previous work. For the sketch-based image retrieval (SBIR), a sketch query is used to retrieve images of objects that belong to the same category, or even with a shape and pose close to the sketch query. The main challenge in the field of SBIR is to obtain efficient cross-domain features for sketch-image similarity measure. For this, besides comparing features extracted from different depth, we additionally compare different training approaches (some novel) for the ConvNets applied to sketches and images. Eventually, our best SBIR system achieves state-of-the-art results on the sketchy database (close to 40% recall at k = 1).

Query-by-example music information retrieval by score-based genre prediction and similarity measure

A topic of music information retrieval (MIR) field is query-by-example (QBE), which searches a popular music dataset using a user-provided query and aims to find the target song. Since this type of MIR has been generally used in online systems, retrieval time is also as important as accuracy. In this paper, we propose a QBE-based MIR system and investigate the impact of automatic music genre prediction on the performance of it, specifically on perspective of accuracy-time trade-off, using a score-based genre prediction method as well as similarity measures. The proposed system is evaluated on a dataset containing 6000 music pieces from six musical genres, and we show that how much improvement on the performance can be achieved in terms of accuracy and retrieval time, compared with a typical QBE-based MIR system that uses only similarity measures to find the user-desired song.

A query-by-example music retrieval system using feature and decision fusion

An attractive topic of Music Information Retrieval (MIR) is focused on query-by-example (QBE), which receives a user-provided query and aims to find the target song from an associated music dataset. In this paper, we use feature and decision fusion techniques to develop a two-stage accurate and rapid QBE based MIR system. For this purpose, a proposed diverse ensemble of recognizers automatically recognizes the genre of the query in first stage. This diversity is yielded through feature extraction over different frequency bands followed by feature fusion to train the recognizers, and then a decision fusion technique fuses the individual results obtained by members of ensemble. Second stage measures similarity between query and other contents of dataset having the same genre with the query to find the target song. To accomplish this, a distance measure that here is Kullback---Leibler divergence is utilized. In this stage, a genre-adaptive feature extraction method is proposed, and features are also fused by a feature fusion technique. The effectiveness of the feature and decision fusion techniques in our two-stage system (genre recognition; song retrieval) is evaluated experimentally that shows a significant improvement in terms of accuracy and retrieval time in comparison with a system for which those techniques are not applied.

Use of Tonic and Raga as Indices for a Query by Example based Music Information Retrieval System

Background and Objectives: A Music Information Retrieval Systems deal with retrieving music from a corpus based on user’s input. Query by Example (QBE) and Query By Humming (QBH) are the two content baste music information retrieval systems. Methods/Statistical Analysis: In this paper, a Query By Example (QBE) based Music Information Retrieval system is proposed, where the characteristics of Carnatic music are used as indices for identifying similar matches to an input song. The system begins by introducing a novel focused crawler mechanism that explores the World Wide Web in a methodical manner to harvest music for the database required. The crawled music is then downloaded, and a local repository of songs is created. Carnatic music features like Tonic and Raga are extracted from the crawled songs and these songs are indexed into the database. Tonic identification is a challenging problem and in this work, a new algorithm for estimation of tonic is designed. Using the Tonic, the Raga of the song is identified and these features are used in a modified, optimized version of the Multi-key hashing technique to index the songs and improve the speed of retrieval. During a user query, which is by example, the same features are extracted, compared with the features of the songs indexed in the database, based on the search option provided by the user, and the best matching songs are retrieved, by proposing a new ranking algorithm based on Raga and Tonic similarity. Findings: The proposed system recognized the Tonic of the song with accuracy while the Raga of the song is recognized if the input music piece is free of Gamaka. Application/ Improvements: Music information retrieval system for entertainment, therapy and helps in content based retrieval of music and the system could be improved to handle Gamakas in the input music piece.

Query-by-example surgical activity detection.

Easy acquisition of surgical data opens many opportunities to automate skill evaluation and teaching. Current technology to search tool motion data for surgical activity segments of interest is limited by the need for manual pre-processing, which can be prohibitive at scale. We developed a content-based information retrieval method, query-by-example (QBE), to automatically detect activity segments within surgical data recordings of long duration that match a query. The example segment of interest (query) and the surgical data recording (target trial) are time series of kinematics. Our approach includes an unsupervised feature learning module using a stacked denoising autoencoder (SDAE), two scoring modules based on asymmetric subsequence dynamic time warping (AS-DTW) and template matching, respectively, and a detection module. A distance matrix of the query against the trial is computed using the SDAE features, followed by AS-DTW combined with template scoring, to generate a ranked list of candidate subsequences (substrings). To evaluate the quality of the ranked list against the ground-truth, thresholding conventional DTW distances and bipartite matching are applied. We computed the recall, precision, F1-score, and a Jaccard index-based score on three experimental setups. We evaluated our QBE method using a suture throw maneuver as the query, on two tool motion datasets (JIGSAWS and MISTIC-SL) captured in a training laboratory. We observed a recall of 93, 90 and 87% and a precision of 93, 91, and 88% with same surgeon same trial (SSST), same surgeon different trial (SSDT) and different surgeon (DS) experiment setups on JIGSAWS, and a recall of 87, 81 and 75% and a precision of 72, 61, and 53% with SSST, SSDT and DS experiment setups on MISTIC-SL, respectively. We developed a novel, content-based information retrieval method to automatically detect multiple instances of an activity within long surgical recordings. Our method demonstrated adequate recall across different complexity datasets and experimental conditions.

Arabic Audio News Retrieval System Using Dependent Speaker Mode, Mel Frequency Cepstral Coefficient and Dynamic Time Warping Techniques

Recently, audio data has increasingly becomes one of the prevalent source of information, especially after the exponential growth of using Internet, digital libraries systems and digital mobile devices. The currently massive amount of audio data stimulates working on developing custom audio retrieval tools to facilitate the audio retrieval tasks. The most familiar audio retrieval systems are based on searching using keyword, title or authors. This study presents the feasibility of using MEL Frequency Cepstral Coefficients (MFCCs) to extract features and Dynamic Time Warping (DTW) to compare the test patterns for Arabic audio news. The study proposes and implements architecture for content based audio retrieval system that is dedicated for the Arabic Audio News. The proposed architecture (ARANEWS) utilizes automatic speech recognition for isolated Arabic keyword speech mode; template based automatic speech recognition approach, MFCCs and DTW. ARANEWS presents a style of retrieval system that based on modeling signal waves and measuring the similarity between features that are extracted from spoken queries and spoken keywords. One of the major components that compose ARANEWS system is feature Database (ARANEWSDB). ARANEWSDB stores the extracted features (MFCCs) from the spoken keywords that are prepared to retrieve Arabic audio news. ARANEWS supports using Query by Humming (QBH) and Query by Example (QBE) instead of using query by text.

Fuzzy Multilevel Graph Embedding for Recognition, Indexing and Retrieval of Graphic Document Images

This thesis addresses the problem of lack of efficient computational tools for graph based structural pattern recognition approaches and proposes to exploit computational strength of statistical pattern recognition. It has two fold contributions. The first contribution is a new method of explicit graph embedding. The proposed graph embedding method exploits multilevel analysis of graph for extracting graph level information, structural level information and elementary level information from graphs. It embeds this information into a numeric feature vector. The method employs fuzzy overlapping trapezoidal intervals for addressing the noise sensitivity of graph representations and for minimizing the information loss while mapping from continuous graph space to discrete vector space. The method has unsupervised learning abilities and is capable of automatically adapting its parameters to underlying graph dataset. The second contribution is a framework for automatic indexing of graph repositories for graph retrieval and subgraph spotting. This framework exploits explicit graph embedding for representing the cliques of order 2 by numeric feature vectors, together with classification and clustering tools for automatically indexing a graph repository. It does not require a labeled learning set and can be easily deployed to a range of application domains, offering ease of query by example (QBE) and granularity of focused retrieval.

On-device mobile visual location recognition by using panoramic images and compressed sensing based visual descriptors.

Mobile Visual Location Recognition (MVLR) has attracted a lot of researchers' attention in the past few years. Existing MVLR applications commonly use Query-by-Example (QBE) based image retrieval principle to fulfill the location recognition task. However, the QBE framework is not reliable enough due to the variations in the capture conditions and viewpoint changes between the query image and the database images. To solve the above problem, we make following contributions to the design of a panorama based on-device MVLR system. Firstly, we design a heading (from digital compass) aware BOF (Bag-of-features) model to generate the descriptors of panoramic images. Our approach fully considers the characteristics of the panoramic images and can facilitate the panorama based on-device MVLR to a large degree. Secondly, to search high dimensional visual descriptors directly on mobile devices, we propose an effective bilinear compressed sensing based encoding method. While being fast and accurate enough for on-device implementation, our algorithm can also reduce the memory usage of projection matrix significantly. Thirdly, we also release a panoramas database as well as a set of test panoramic quires which can be used as a new benchmark to facilitate further research in the area. Experimental results prove the effectiveness of the proposed methods for on-device MVLR applications.

Multiobjective Time Series Matching for Audio Classification and Retrieval

Seeking sound samples in a massive database can be a tedious and time consuming task. Even when metadata are available, query results may remain far from the timbre expected by users. This problem stems from the nature of query specification, which does not account for the underlying complexity of audio data. The Query By Example (QBE) paradigm tries to tackle this shortcoming by finding audio clips similar to a given sound example. However, it requires users to have a well-formed soundfile of what they seek, which is not always a valid assumption. Furthermore, most audio-retrieval systems rely on a single measure of similarity, which is unlikely to convey the perceptual similarity of audio signals. We address in this paper an innovative way of querying generic audio databases by simultaneously optimizing the temporal evolution of multiple spectral properties. We show how this problem can be cast into a new approach merging multiobjective optimization and time series matching, called MultiObjective Time Series (MOTS) matching. We formally state this problem and report an efficient implementation. This approach introduces a multidimensional assessment of similarity in audio matching. This allows to cope with the multidimensional nature of timbre perception and also to obtain a set of efficient propositions rather than a single best solution. To demonstrate the performances of our approach, we show its efficiency in audio classification tasks. By introducing a selection criterion based on the hypervolume dominated by a class, we show that our approach outstands the state-of-art methods in audio classification even with a few number of features. We demonstrate its robustness to several classes of audio distortions. Finally, we introduce two innovative applications of our method for sound querying.

Searching for images by video

Image retrieval based on the query-by-example (QBE) principle is still not reliable enough, largely because of the likely variations in the capture conditions (e.g. light, blur, scale, occlusion) and viewpoint between the query image and the images in the collection. In this paper, we propose a framework in which this problem is explicitly addressed to improve the reliability of QBE-based image retrieval. We aim at the use scenario involving the user capturing the query object by his/her mobile device and requesting information augmenting the query from the database. Reliability improvement is achieved by allowing the user to submit not a single image but a short video clip as a query. Since a video clip may combine object or scene appearances captured from different viewpoints and under different conditions, the rich information contained therein can be exploited to discover the proper query representation and to improve the relevance of the retrieved results. The experimental results show that video-based image retrieval (VBIR) is significantly more reliable than the retrieval using a single image as query. Furthermore, to make the proposed framework deployable in a practical mobile image retrieval system, where realtime query response is required, we also propose the priority queue-based feature description scheme and cache-based bi-quantization algorithm for an efficient parallel implementation of the VBIR concept.