Multiple Fundamental Frequency Estimation Research Articles

Multipitch estimation based on the iterative detection and separation of note events from single-channel polyphonic recordings.

Multiple fundamental frequency estimation has been extensively used in applications such as melody extraction, music transcription, instrument identification, and source separation. This paper presents an approach based on the iterative detection and extraction of note events, which are considered to be harmonic sounds characterised by a continuous pitch trajectory. Note events are assumed to be associated with musical notes being played by a single instrument, and their pitch trajectories are iteratively estimated. In every iteration, the pitch contour of the predominant note event is selected from a set of pitch estimates and used to separate its spectral energy from the input mixture in order to obtain a residual signal, which is then used as input in the next iteration. This iterative process stops when the energy of the residual is below a significance threshold. The pitch trajectories of all detected note events are then revised and reassembled to form the final set of pitch estimates for the original audio input. Evaluation of performance is conducted in different scenarios to show the potential of the proposed system, both in terms of its accuracy, and also as an initial stage in other complex tasks, such as note tracking and multipitch streaming.

Dagstuhl ChoirSet: A Multitrack Dataset for MIR Research on Choral Singing

Choral singing is a central part of musical cultures across the world, yet many facets of this widespread form of polyphonic singing are still to be explored. Music information retrieval (MIR) research on choral singing benefits from multitrack recordings of the individual singing voices. However, there exist only few publicly available multitrack datasets on polyphonic singing. In this paper, we present Dagstuhl ChoirSet (DCS), a multitrack dataset of a cappella choral music designed to support MIR research on choral singing. The dataset includes recordings of an amateur vocal ensemble performing two choir pieces in full choir and quartet settings. The audio data was recorded during an MIR seminar at Schloss Dagstuhl using different close-up microphones to capture the individual singers’ voices. In this article, we give detailed insights into all stages of creating DCS: recording process, data preparation, generation of annotations as well as development of suitable interfaces for publicly accessing and reusing the data. Furthermore, we demonstrate the potential of the dataset for MIR research by discussing case studies on choral intonation assessment and multiple fundamental frequency (F0) estimation.

Robust Real-Time Music Transcription with a Compositional Hierarchical Model.

The paper presents a new compositional hierarchical model for robust music transcription. Its main features are unsupervised learning of a hierarchical representation of input data, transparency, which enables insights into the learned representation, as well as robustness and speed which make it suitable for real-world and real-time use. The model consists of multiple layers, each composed of a number of parts. The hierarchical nature of the model corresponds well to hierarchical structures in music. The parts in lower layers correspond to low-level concepts (e.g. tone partials), while the parts in higher layers combine lower-level representations into more complex concepts (tones, chords). The layers are learned in an unsupervised manner from music signals. Parts in each layer are compositions of parts from previous layers based on statistical co-occurrences as the driving force of the learning process. In the paper, we present the model’s structure and compare it to other hierarchical approaches in the field of music information retrieval. We evaluate the model’s performance for the multiple fundamental frequency estimation. Finally, we elaborate on extensions of the model towards other music information retrieval tasks.

Guitar Chords Classification Using Uncertainty Measurements of Frequency Bins

This paper presents a method to perform chord classification from recorded audio. The signal harmonics are obtained by using the Fast Fourier Transform, and timbral information is suppressed by spectral whitening. A multiple fundamental frequency estimation of whitened data is achieved by adding attenuated harmonics by a weighting function. This paper proposes a method that performs feature selection by using a thresholding of the uncertainty of all frequency bins. Those measurements under the threshold are removed from the signal in the frequency domain. This allows a reduction of 95.53% of the signal characteristics, and the other 4.47% of frequency bins are used as enhanced information for the classifier. An Artificial Neural Network was utilized to classify four types of chords: major, minor, major 7th, and minor 7th. Those, played in the twelve musical notes, give a total of 48 different chords. Two reference methods (based on Hidden Markov Models) were compared with the method proposed in this paper by having the same database for the evaluation test. In most of the performed tests, the proposed method achieved a reasonably high performance, with an accuracy of 93%.

Multipitch estimation using judge-based model

Abstract Multipitch estimation, also known as multiple fundamental frequency (F0) estimation, is an important part of the Music Information Retrieval (MIR) field. Although there have been many different approaches proposed, none of them has ever exceeded the abilities of a trained musician. In this work, an iterative cancellation method is analysed, being applied to three different sound representations - salience spectrum obtained using Constant-Q Transform, cepstrum and enhanced autocorrelation result. Real-life recordings of different musical instruments are used as a database and the parameters of the solution are optimized using a simple yet effective metaheuristic approach - the Luus-Jaakola algorithm. The presented approach results in 85% efficiency on the test database.

Automatic Transcription of Polyphonic Piano Music Using Genetic Algorithms, Adaptive Spectral Envelope Modeling, and Dynamic Noise Level Estimation

This paper presents a new method for multiple fundamental frequency (F0) estimation on piano recordings. We propose a framework based on a genetic algorithm in order to analyze the overlapping overtones and search for the most likely F0 combination. The search process is aided by adaptive spectral envelope modeling and dynamic noise level estimation: while the noise is dynamically estimated, the spectral envelope of previously recorded piano samples (internal database) is adapted in order to best match the piano played on the input signals and aid the search process for the most likely combination of F0s. For comparison, several state-of-the-art algorithms were run across various musical pieces played by different pianos and then compared using three different metrics. The proposed algorithm ranked first place on Hybrid Decay/Sustain Score metric, which has better correlation with the human hearing perception and ranked second place on both onset-only and onset–offset metrics. A previous genetic algorithm approach is also included in the comparison to show how the proposed system brings significant improvements on both quality of the results and computing time.

Multiple fundamental frequency estimation based on sparse representations in a structured dictionary

Automatic transcription of polyphonic music is an important task in audio signal processing, which involves identifying the fundamental frequencies (pitches) of several notes played at a time. Its difficulty stems from the fact that harmonics of different notes tend to overlap, especially in western music. This causes a problem in assigning the harmonics to their true fundamental frequencies, and in deducing spectra of several notes from their sum. We present here a multi-pitch estimation algorithm based on sparse representations in a structured dictionary, suitable for the spectra of music signals. In the vectors of this dictionary, most of the elements are forced to be zero except the elements that represent the fundamental frequencies and their harmonics. Thanks to the structured dictionary, the algorithm does not require a diverse or a large dataset for training and is computationally more efficient than alternative methods. The performance of the proposed structured dictionary transcription system is empirically examined, and its advantage is demonstrated compared to alternative dictionary learning methods.

A Compositional Automatic Music Transcription System for Computersynthesized Music

Automatic music transcription can be widely used for content-based music retrieval, low rate coding and automatic musical accompaniment system and so on. Multiple fundamental frequency estimation, or Multiple-F0 estimation, is one of the most important problems in automatic music transcription, but it has not been well resolved up to now. This paper presents a machine learning methods using harmonic matching and iterative deletion for computer-synthesized music specifically to Multiple-F0 estimation and builds an efficient automatic music transcription system. Computersynthesized music is almost free from similar instruments of the differences between different individuals, so it is a good research object. It can be shown in this paper that the experimental results indicate that this method has very good recognition results.

Automatic Transcription of Guitar Chords and Fingering From Audio

This paper proposes a method for extracting the fingering configurations automatically from a recorded guitar performance. 330 different fingering configurations are considered, corresponding to different versions of the major, minor, major 7th, and minor 7th chords played on the guitar fretboard. The method is formulated as a hidden Markov model, where the hidden states correspond to the different fingering configurations and the observed acoustic features are obtained from a multiple fundamental frequency estimator that measures the salience of a range of candidate note pitches within individual time frames. Transitions between consecutive fingerings are constrained by a musical model trained on a database of chord sequences, and a heuristic cost function that measures the physical difficulty of moving from one configuration of finger positions to another. The method was evaluated on recordings from the acoustic, electric, and the Spanish guitar and clearly outperformed a non-guitar-specific reference chord transcription method despite the fact that the number of chords considered here is significantly larger.

Efficient methods for joint estimation of multiple fundamental frequencies in music signals

This study presents efficient techniques for multiple fundamental frequency estimation in music signals. The proposed methodology can infer harmonic patterns from a mixture considering interactions with other sources and evaluate them in a joint estimation scheme. For this purpose, a set of fundamental frequency candidates are first selected at each frame, and several hypothetical combinations of them are generated. Combinations are independently evaluated, and the most likely is selected taking into account the intensity and spectral smoothness of its inferred patterns. The method is extended considering adjacent frames in order to smooth the detection in time, and a pitch tracking stage is finally performed to increase the temporal coherence. The proposed algorithms were evaluated in MIREX contests yielding state of the art results with a very low computational burden.

Maximum A Posteriori Probability Multiple-Pitch Tracking Using the Harmonic Model

In this paper, a new method for multiple fundamental frequency estimation for speech and music signals is proposed. Applications of audio and speech processing include many well-reviewed algorithms for estimating the fundamental frequency of monophonic speech and music signals. In the case of polyphonic signals, it is more difficult to successfully estimate each of the fundamental frequencies, as reflected by the dearth of existing methods addressing this problem. In this paper, a new method based on the combination of the maximum likelihood and maximum a posteriori probability criteria is derived for fundamental frequencies tracking where each one of the fundamental frequencies is modeled by a first-order Markov process. The dominant signal is modeled as a harmonic source with unknown deterministic amplitudes, while the remaining signals, including other harmonic signals, are modeled as Gaussian interference sources with an unknown covariance matrix. After estimation of the dominant source, it is removed from the signal by projection of the signal into the null subspace spanned by the estimated signal. This procedure is iterated for all the harmonic sources in the data. The algorithm is tested with speech, music, and synthetic signals where in each case, two harmonic sources of the same kind were mixed. The performance of the proposed algorithm is evaluated and compared to an existing reference method in terms of gross-error-rate as a function of signal-to-interference ratio.

Multiple Fundamental Frequency Estimation by Modeling Spectral Peaks and Non-Peak Regions

This paper presents a maximum-likelihood approach to multiple fundamental frequency (F0) estimation for a mixture of harmonic sound sources, where the power spectrum of a time frame is the observation and the F0s are the parameters to be estimated. When defining the likelihood model, the proposed method models both spectral peaks and non-peak regions (frequencies further than a musical quarter tone from all observed peaks). It is shown that the peak likelihood and the non-peak region likelihood act as a complementary pair. The former helps find F0s that have harmonics that explain peaks, while the latter helps avoid F0s that have harmonics in non-peak regions. Parameters of these models are learned from monophonic and polyphonic training data. This paper proposes an iterative greedy search strategy to estimate F0s one by one, to avoid the combinatorial problem of concurrent F0 estimation. It also proposes a polyphony estimation method to terminate the iterative process. Finally, this paper proposes a postprocessing method to refine polyphony and F0 estimates using neighboring frames. This paper also analyzes the relative contributions of different components of the proposed method. It is shown that the refinement component eliminates many inconsistent estimation errors. Evaluations are done on ten recorded four-part J. S. Bach chorales. Results show that the proposed method shows superior F0 estimation and polyphony estimation compared to two state-of-the-art algorithms.

Error correction and accuracy scaling for a joint subtraction multiple fundamental frequency estimation algorithm

This talk presents an improved multiple fundamental frequency (F0) estimation algorithm, which is loosely based on a previously developed joint subtraction algorithm [A. Klapuri, Proc. 7th Int. Conf. on Music Information Retrieval, 216–221 (2006)]. The improved algorithm uses some auditory scene analysis techniques to supplement a maximum likelihood based F0 estimator to predict the best combination of F0s. The accuracy of this base algorithm is then improved by use of error detection/correction. Error rates of the algorithm were collected using random mixes of sounds gathered from the University of Iowa Theremin database of musical instrument samples, as well as mixes of continuous audio. The performance of both the base algorithm and the improved algorithm will be presented together. It was found that the improved algorithm decreases error rates by as much as 50%, compared to the base algorithm with minimal loss in computational efficiency. The trade-offs between computational efficiency and accuracy, tested via a tunable parameter built into the algorithm, will be discussed. Trade-offs between computational costs and accuracy as a function of the number of F0s will also be discussed.

Algorithm for separating vocals from polyphonic music

An algorithm for the separation of vocals from polyphonic music is described. The algorithm consists of two stages, which first estimate the predominant melody line, and then the sinusoidal modeling parameters corresponding to the melody line. The melody line is estimated using a hidden Markov model where the output of a multiple fundamental frequency estimator is used as a feature set. The states of the hidden Markov model correspond to musical notes having different fundamental frequencies, and the state transition probabilities are determined by a musicological model. The sinusoidal modeling stage estimates the frequency, amplitude, and phase of each overtone of the predominant melody line in each frame. The sinusoidal modeling stage can also include mechanisms which reduce the effect of interfering sound sources on the estimated parameters. The resulting separation algorithm is independent of singer identity, musical genre, or instrumentation. Simulation experiments on real polyphonic music show that the algorithm enables separation of vocals from the accompaniment, providing robust results on various musical genres.

Multiple fundamental frequency estimation assisted by audio watermarking

The success of computer segregation of sound sources from a single‐channel mixture often relies on the estimation of multiple fundamental frequencies. Instead of solving the problem directly, this presented work utilizes audio watermarks to assist sound source segregation. Here, individual sources are assumed to be available for watermark embedding before mixing. Each source’s short‐time spectral peaks are aligned to frequency gridpoints labeled with a digital ID [Liu and Smith, ‘‘Audio Watermarking Based on Sinusoidal Analysis and Synthesis,’’ International Symposium on Musical Acoustics, Nara, Japan (2004)]. A modified sound source is sinusoidally synthesized afterwards. In this sense, each source is embedded with a watermark. In an experiment, two sound sources are watermarked with two sets of gridpoints P and Q, respectively. Points in P and Q interleave and are spaced 10 cents of a semitone apart. The watermarked sources are then linearly added. To demix them, sinusoidal trajectories found in the mix...

Multiple fundamental frequency estimation based on harmonicity and spectral smoothness

A new method for estimating the fundamental frequencies of concurrent musical sounds is described. The method is based on an iterative approach, where the fundamental frequency of the most prominent sound is estimated, the sound is subtracted from the mixture, and the process is repeated for the residual signal. For the estimation stage, an algorithm is proposed which utilizes the frequency relationships of simultaneous spectral components, without assuming ideal harmonicity. For the subtraction stage, the spectral smoothness principle is proposed as an efficient new mechanism in estimating the spectral envelopes of detected sounds. With these techniques, multiple fundamental frequency estimation can be performed quite accurately in a single time frame, without the use of long-term temporal features. The experimental data comprised recorded samples of 30 musical instruments from four different sources. Multiple fundamental frequency estimation was performed for random sound source and pitch combinations. Error rates for mixtures ranging from one to six simultaneous sounds were 1.8%, 3.9%, 6.3%, 9.9%, 14%, and 18%, respectively. In musical interval and chord identification tasks, the algorithm outperformed the average of ten trained musicians. The method works robustly in noise, and is able to handle sounds that exhibit inharmonicities. The inharmonicity factor and spectral envelope of each sound is estimated along with the fundamental frequency.

Separation of harmonic sounds using multipitch analysis and linear models for the overtone series

A signal processing method for the separation of concurrent harmonic sounds is described. The method is based on a two-stage approach. First, a multiple fundamental frequency estimator is applied to find initial sound parameters which are reliable, but inaccurate and static. Second, time-varying sinusoidal parameters are estimated in an iterative algorithm. The harmonic structure is retained by keeping the frequency ratio of overtones constant over time. Overlapping harmonic components are resolved using linear models for the overtone series. In practice, the models retain the spectral envelope continuity of natural sounds. Simulation experiments were carried out using generated test signals, which were random mixtures of two to six notes from recorded natural instruments. The system is able to produce meaningful results in all polyphonies, the quality of separated sounds gradually degrading along with the polyphony. Some denoising algorithms were applied to suppress nonstationary noise component, such as drums in real-world music signals. However, the usability of the system for real musical signals is still quite limited.