Query-by-singing/humming Research Articles

Fast Query-by-Singing/Humming System That Combines Linear Scaling and Quantized Dynamic Time Warping Algorithm

We newly propose a query-by-singing/humming (QbSH) system considering both the preclassification and multiple classifier-based method by combining linear scaling (LS) and quantized dynamic time warping (QDTW) algorithm in order to enhance both the matching accuracy and processing speed. This is appropriate for the QbSH of high speed in the huge distributed server environment. This research is novel in the following three ways. First, the processing speed of the QDTW is generally much slower than the LS method. So, we perform the QDTW matching only in case that the matching distance by LS algorithm is smaller than predetermined threshold, by which the entire processing time is reduced while the matching accuracy is maintained. Second, we use the different measurement method of matching distance in LS algorithm by considering the characteristics of reference database. Third, we combine the calculated distances of LS and QDTW algorithms based on score level fusion in order to enhance the matching accuracy. The experimental results with the 2009 MIR-QbSH corpus and the AFA MIDI 100 databases showed that the proposed method reduced the total searching time of reference data while obtaining the higher accuracy compared to the QDTW.

A Novel Query-by-Singing/Humming Method by Estimating Matching Positions Based on Multi-layered Perceptron

The increase in the number of music files in smart phone and MP3 player makes it difficult to find the music files which people want. So, Query-by-Singing/Humming (QbSH) systems have been developed to retrieve music from a user’s humming or singing without having to know detailed information about the title or singer of song. Most previous researches on QbSH have been conducted using musical instrument digital interface (MIDI) files as reference songs. However, the production of MIDI files is a time-consuming process. In addition, more and more music files are newly published with the development of music market. Consequently, the method of using the more common MPEG-1 audio layer 3 (MP3) files for reference songs is considered as an alternative. However, there is little previous research on QbSH with MP3 files because an MP3 file has a different waveform due to background music and multiple (polyphonic) melodies compared to the humming/singing query. To overcome these problems, we propose a new QbSH method using MP3 files on mobile device. This research is novel in four ways. First, this is the first research on QbSH using MP3 files as reference songs. Second, the start and end positions on the MP3 file to be matched are estimated by using multi-layered perceptron (MLP) prior to performing the matching with humming/singing query file. Third, for more accurate results, four MLPs are used, which produce the start and end positions for dynamic time warping (DTW) matching algorithm, and those for chroma-based DTW algorithm, respectively. Fourth, two matching scores by the DTW and chroma-based DTW algorithms are combined by using PRODUCT rule, through which a higher matching accuracy is obtained. Experimental results with AFA MP3 database show that the accuracy (Top 1 accuracy of 98%, with an MRR of 0.989) of the proposed method is much higher than that of other methods. We also showed the effectiveness of the proposed system on consumer mobile device.

Implementation of a practical query-by-singing/humming (QbSH) system and its commercial applications

In this paper, a practical query-bysinging/humming (QbSH) system is proposed that uses polyphonic music tracks such as MP3 and AAC files to create the reference database (DB) unlike conventional QbSH systems. To create the reference DB, we propose a method for melody extraction from polyphonic music signals based on harmonic structure. In addition, we propose a matching engine using modified dynamic time warping (DTW) that uses chroma-scale representation and asymmetric path of DTW to reduce the influence of melody extraction error. We implemented three different prototypes for its commercial applications like smart phone, laptop and karaoke. We evaluated the performance of the proposed practical QbSH system with monophonic and polyphonic music datasets, and confirmed that it has an acceptable performance for commercial applications.

An Extensive Analysis of Query by Singing/Humming System Through Query Proportion

Query by Singing/Humming (QBSH) is a Music Information Retrieval (MIR) system with small audio excerpt as query. The rising availability of digital music stipulates effective music retrieval methods. Further, MIR systems support content based searching for music and requires no musical acquaintance. Current work on QBSH focuses mainly on melody features such as pitch, rhythm, note etc., size of databases, response time, score matching and search algorithms. Even though a variety of QBSH techniques are proposed, there is a dearth of work to analyze QBSH through query excerption. Here, we present an analysis that works on QBSH through query excerpt. To substantiate a series of experiments are conducted with the help of Mel-Frequency Cepstral Coefficients (MFCC), Linear Predictive Coefficients (LPC) and Linear Predictive Cepstral Coefficients (LPCC) to portray the robustness of the knowledge representation. Proposed experiments attempt to reveal that retrieval performance as well as precision diminishes in the snail phase with the growing database size.

An FFT-based fast melody comparison method for query-by-singing/humming systems

Query-by-singing/humming (QBSH) is a promising way to retrieve music recordings based on main melody’s similarity. This paper presents an efficient QBSH method that enables fast melody comparison. In contrast to the most prevalent QBSH method, which measures the distances between note sequences in the time domain, the proposed method performs distance computation in the frequency domain. This is done with the fast Fourier transform, which converts different-length note sequences into equal-dimension vectors via zero padding. The equal dimensionality allows us to compare the vectors using Euclidean distance directly, which avoids performing time-consuming alignment between sequences. To take both efficiency and effectiveness into account, the proposed fast melody comparison method is combined with the dynamic time warping technique into a two-stage sequence matching system. Our experiments show that the proposed system outperforms several existing speed-up DTW-based systems in terms of both efficiency and effectiveness.

Design and Implementation of a Fixed-Mobile Convergent Music Search Engine (FMC-MUSE)

Query by Singing/Humming (QBSH) is a most natural way for music search. A music search system can help music finders search songs by matching a part of melody by singing or humming. Many music information retrieval techniques have been developed to carry out music search for years. On the other hand, thanks to the rapid growth of mobile wireless Internet technologies this decade, music search applications can be implemented on hand-carried devices, such as cellular phones, to conduct music search anytime and anywhere via any available networks, such as Wi-Fi, UMTS, WiMAX to the emerging 3GPP-LTE networks. In the past, little studies had ever been revealed about how to design and implement a lightweight music search engine over a fixed or mobile Internet. In this article, we aim to elaborate a practical skeleton of developing a simple music search engine over fixed or mobile networks--a Fixed-Mobile Convergent Music Search Engine (FMC-MUSE). FMC-MUSE can process music queries by QBSH from fixed or mobile clients and return a dataset containing the search results and meta-info back to music finders via ubiquitous networks.

오디오 Fingerprint를 이용한 음악인식 연구 동향

Interest in music recognition has been growing dramatically after NHN and Daum released their mobile applications for music recognition in 2010. Methods in music recognition based on audio analysis fall into two categories: music recognition using audio fingerprint and Query-by-Singing/Humming (QBSH). While music recognition using audio fingerprint receives music as its input, QBSH involves taking a user-hummed melody. In this paper, research trends are described for music recognition using audio fingerprint, focusing on two methods: one based on fingerprint generation using energy difference between consecutive bands and the other based on hash key generation between peak points. Details presented in the representative papers of each method are introduced.

다성음원 기반 QbSH 시스템을 위한 매칭엔진의 설계 및 구현

This paper proposes a matching engine of query-by-singing/humming (QbSH) system which retrieves the most similar music information by comparing the input data with the extracted feature information from polyphonic music like MP3. The feature sequences transcribed from polyphonic music may have many errors. So, to reduce the influence of errors and improve the performance, the chroma-scale representation, compensation and asymmetric DTW (Dynamic Time Warping) are adopted in the matching engine. The performance of various distance metrics are also investigated in this paper. In our experiment, the proposed QbSH system achieves MRR (Mean Reciprocal Rank) of 0.718 for 1000 singing/humming queries when searching from a database of 450 polyphonic musics.

Robust query-by-singing/humming system against background noise environments

Under background noise environments, the performance of the Query-by-Singing/Humming (QbSH) system is considerably degraded. Since human pitch information is used as a feature vector for the QbSH system, a noise robust pitch estimation algorithm is inevitable. Thus, a novel pitch-estimation method is proposed by integrating temporal-autocorrelation and spectral-salience methods. As a pre-processing block, spectral smoothing is applied to enhance the stationarity of the noisy input signal. To calculate the similarity between the MIDI database and input humming signal, the dynamic time warping (DTW) algorithm is used. Jang's corpus and AURORA2 database are selected as humming and background noise signals, respectively. Compared with the standard pitch estimation algorithm in the ITU-T G.729 speech codec, the proposed pitch estimation method improves the average accuracy by 11.7% for the 0 dB signal-to-noise ratio (SNR) noise case. It also improves top-20 ratio and mean reciprocal rank (MRR) of the proposed QbSH system, on average, by 7.4% and 0.13, respectively.

A General Framework of Progressive Filtering and Its Application to Query by Singing/Humming

This paper presents the mathematical formulation and design methodology of progressive filtering (PF) for multimedia information retrieval, and discusses its application to the so-called query by singing/humming (QBSH), or more formally, melody recognition. The concept of PF and the corresponding dynamic programming-based design method are applicable to large multimedia retrieval systems for striking a balance between efficiency (in terms of response time) and effectiveness (in terms of recognition rate). The application of the proposed PF to a five-stage QBSH system is reported, and the experimental results demonstrate the feasibility of the proposed approach.