Polyphonic Signals Research Articles

Efficient bandwidth extension of musical signals using a differentiable harmonic plus noise model

The task of bandwidth extension addresses the generation of missing high frequencies of audio signals based on knowledge of the low-frequency part of the sound. This task applies to various problems, such as audio coding or audio restoration. In this article, we focus on efficient bandwidth extension of monophonic and polyphonic musical signals using a differentiable digital signal processing (DDSP) model. Such a model is composed of a neural network part with relatively few parameters trained to infer the parameters of a differentiable digital signal processing model, which efficiently generates the output full-band audio signal. We first address bandwidth extension of monophonic signals, and then propose two methods to explicitly handle polyphonic signals. The benefits of the proposed models are first demonstrated on monophonic and polyphonic synthetic data against a baseline and a deep-learning-based ResNet model. The models are next evaluated on recorded monophonic and polyphonic data, for a wide variety of instruments and musical genres. We show that all proposed models surpass a higher complexity deep learning model for an objective metric computed in the frequency domain. A MUSHRA listening test confirms the superiority of the proposed approach in terms of perceptual quality.

Nonnegative Matrix Factorization With Basis Clustering Using Cepstral Distance Regularization

One successful approach for audio source separation involves applying nonnegative matrix factorization (NMF) to a magnitude spectrogram regarded as a nonnegative matrix. This can be interpreted as approximating the observed spectra at each time frame as the linear sum of the basis spectra scaled by time-varying amplitudes. This paper deals with the problem of the unsupervised instrument-wise source separation of polyphonic signals based on an extension of the NMF approach. We focus on the fact that each piece of music is typically played on a handful of musical instruments, which allows us to assume that the spectra of the underlying audio events in a polyphonic signal can be grouped into a reasonably small number of clusters in the mel-frequency cepstral coefficient (MFCC) domain. Based on this assumption, we propose formulating factorization of a magnitude spectrogram and clustering of the basis spectra in the MFCC domain as a joint optimization problem and derive a novel optimization algorithm based on the majorization–minimization principle. Experimental results revealed that our method was superior to a two-stage algorithm that consists of performing factorization followed by clustering the basis spectra, thus showing the advantage of the joint optimization approach.

Real-Time Audio-to-Score Alignment of Music Performances Containing Errors and Arbitrary Repeats and Skips

This paper discusses real-time alignment of audio signals of music performance to the corresponding score (a.k.a. score following) which can handle tempo changes, errors and arbitrary repeats and/or skips (repeats/skips) in performances. This type of score following is particularly useful in automatic accompaniment for practices and rehearsals, where errors and repeats/skips are often made. Simple extensions of the algorithms previously proposed in the literature are not applicable in these situations for scores of practical length due to the problem of large computational complexity. To cope with this problem, we present two hidden Markov models of monophonic performance with errors and arbitrary repeats/skips, and derive efficient score-following algorithms with an assumption that the prior probability distributions of score positions before and after repeats/skips are independent from each other. We confirmed real-time operation of the algorithms with music scores of practical length (around 10000 notes) on a modern laptop and their tracking ability to the input performance within 0.7 s on average after repeats/skips in clarinet performance data. Further improvements and extension for polyphonic signals are also discussed.

Percussive/harmonic sound separation by non-negative matrix factorization with smoothness/sparseness constraints

In this paper, unsupervised learning is used to separate percussive and harmonic sounds from monaural non-vocal polyphonic signals. Our algorithm is based on a modified non-negative matrix factorization (NMF) procedure that no labeled data is required to distinguish between percussive and harmonic bases because information from percussive and harmonic sounds is integrated into the decomposition process. NMF is performed in this process by assuming that harmonic sounds exhibit spectral sparseness (narrowband sounds) and temporal smoothness (steady sounds), whereas percussive sounds exhibit spectral smoothness (broadband sounds) and temporal sparseness (transient sounds). The evaluation is performed using several real-world excerpts from different musical genres. Comparing the developed approach to three current state-of-the art separation systems produces promising results.

Cascaded Long Term Prediction for Enhanced Compression of Polyphonic Audio Signals

Audio compression systems exploit periodicity in signals to remove inter-frame redundancies via the long term prediction (LTP) tool. This simple tool capitalizes on the periodic component of the waveform by selecting a past segment as the basis for prediction of the current frame. However, most audio signals are polyphonic in nature, containing a mixture of several periodic components. While such polyphonic signals may themselves be periodic with overall period equaling the least common multiple of the individual component periods, the signal rarely remains sufficiently stationary over the extended period, rendering the LTP tool suboptimal. Instead of seeking a past segment that represents a “compromise” for incompatible component periods, we propose a more complex filter that predicts every periodic component of the signal from its immediate history, and this is achieved by cascading LTP filters, each corresponding to individual periodic component. We also propose a recursive “divide and conquer” technique to estimate parameters of all the LTP filters. We then demonstrate the effectiveness of cascaded LTP in two distinct settings of the ultra low delay Bluetooth Subband Codec and the MPEG Advanced Audio Coding (AAC) standard. In MPEG AAC, we specifically adapt the cascaded LTP parameter estimation to take into account the perceptual distortion criteria, and also propose a low decoder complexity variant. Objective and subjective results for all the settings validate the effectiveness of the proposal on a variety of polyphonic signals.

Constrained non-negative sparse coding using learnt instrument templates for realtime music transcription

In this paper, we present a realtime signal decomposition method with single-pitch and harmonicity constrains using instrument specific information. Although the proposed method is designed for monophonic music transcription, it can be used as a candidate selection technique in combination with other realtime transcription methods to address polyphonic signals. The harmonicity constraint is particularly beneficial for automatic transcription because, in this way, each basis can define a single pitch. Furthermore, restricting the model to have a single-nonzero gain at each frame has been shown to be a very suitable constraint when dealing with monophonic signals. In our method, both harmonicity and single-nonzero gain constraints are enforced in a deterministic manner. A realtime factorization procedure based on Non-negative sparse coding (NNSC) with Beta-divergence and fixed basis functions is proposed. In this paper, the basis functions are learned using a supervised process to obtain spectral patterns for different musical instruments. The proposed method has been tested for music transcription of both monophonic and polyphonic signals and has been compared with other state-of-the-art transcription methods, and in these tests, the proposed method has obtained satisfactory results in terms of accuracy and runtime.

Monophonic constrained non-negative sparse coding using instrument models for audio separation and transcription of monophonic source-based polyphonic mixtures

In this paper we propose a monophonic constrained signal decomposition model applied to polyphonic signals composed of several monophonic sources from different musical instruments. The harmonic constraint is particularly effective for tonal instruments because each note is associated with a unique basis. The monophonic constraint is implemented by enforcing single-non-zero gains per instrument in the factorization process. The proposed method uses previously trained instrument models with a supervised procedure. Both constraints (harmonic and monophonic) are implemented in a deterministic manner. The proposed method has been tested for two audio signal applications, Sound Source Separation and Automatic Music Transcription. Comparison with other state-of-the-art methods using a dataset of polyphonic mixtures composed of monophonic sources has produced competitive and promising results.

A Sub-Band Approach to Modification of Musical Transients

Abstract The transient modifier is a type of audio effect that changes the level of the transient parts in a musical signal while leaving the steady-state parts unchanged. This article presents a high-performance algorithm for transient detection and modification, one that is capable of modifying transients in polyphonic or multi-voiced signals, and capable of modifying both hard (percussive) and soft (non-percussive) transients. The detection and modification of transients are performed in the frequency-domain using a sub-band approach. Detection is based on both phase and energy information using an adaptive threshold, and modification is carried out independently at each sub-band. The performance of the proposed sub-band approach was compared with other transient-modification algorithms using subjective listening tests. We show that the sub-band approach with adaptive threshold mostly outperforms other approaches.

Maximum A Posteriori Probability Multiple-Pitch Tracking Using the Harmonic Model

In this paper, a new method for multiple fundamental frequency estimation for speech and music signals is proposed. Applications of audio and speech processing include many well-reviewed algorithms for estimating the fundamental frequency of monophonic speech and music signals. In the case of polyphonic signals, it is more difficult to successfully estimate each of the fundamental frequencies, as reflected by the dearth of existing methods addressing this problem. In this paper, a new method based on the combination of the maximum likelihood and maximum a posteriori probability criteria is derived for fundamental frequencies tracking where each one of the fundamental frequencies is modeled by a first-order Markov process. The dominant signal is modeled as a harmonic source with unknown deterministic amplitudes, while the remaining signals, including other harmonic signals, are modeled as Gaussian interference sources with an unknown covariance matrix. After estimation of the dominant source, it is removed from the signal by projection of the signal into the null subspace spanned by the estimated signal. This procedure is iterated for all the harmonic sources in the data. The algorithm is tested with speech, music, and synthetic signals where in each case, two harmonic sources of the same kind were mixed. The performance of the proposed algorithm is evaluated and compared to an existing reference method in terms of gross-error-rate as a function of signal-to-interference ratio.

Missing Data Imputation for Time-Frequency Representations of Audio Signals

With the recent attention towards audio processing in the time-frequency domain we increasingly encounter the problem of missing data within that representation. In this paper we present an approach that allows us to recover missing values in the time-frequency domain of audio signals. The presented approach is able to deal with real-world polyphonic signals by operating seamlessly even in the presence of complex acoustic mixtures. We demonstrate that this approach outperforms generic missing data approaches, and we present a variety of situations that highlight its utility.

Constrained filter encoding of polyphonic signals

Signals of different channels (c1-cN) are combined into one mono signal (x). A set of adaptive filters, preferably one for each channel (c-cN), is derived in a respective filter adaptation unit (30:1-30:N). when an adaptive filter is applied to the mono signal (x) it reconstructs the signal of the respective channel (c1-cN) under a perceptual constraint. The perceptual constraint is a gain and/or shape constraint. The gain constraint allows the preservation of the relative energy between the channels (c1-cN) while the shape constraint allows more stability by avoiding unnecessary filtering of spectrum nulls. The transmitted parameters are the mono signal (x), in encoded form, and the parameters (p1-pN) of the adaptive filters, preferably also encoded. The receiver reconstructs the signal of the different channels by applying the adaptive filters and possibly some additional post-processing.

APPLICATION OF FAST FILTER BANK FOR TRANSCRIPTION OF POLYPHONIC SIGNALS

In this paper, a novel approach to the transcription of polyphonic signals is investigated. The method makes use of top–down analysis algorithm and bottom–up reconstruction of frequency contents of musical signals using a reference database that contains the spectral profiles of all single notes. Recognition accuracy is enhanced by analyzing the time-frequency contents using a bank of narrow band filters with very sharp transition band. The filters are designed using the Frequency-Response Masking technique. The high selectivity of the filters has enhanced the frequency analysis and increased the accuracy of identification. The method was tested with musical notes generated from an acoustic piano. The results show that for single notes and chords up to four notes, perfect identification can be achieved.

Pitch Histograms in Audio and Symbolic Music Information Retrieval

In order to represent musical content, pitch and timing information is utilized in the majority of existing work in Symbolic Music Information Retrieval (MIR). Symbolic representations such as MIDI allow the easy calculation of such information and its manipulation. In contrast, most of the existing work in Audio MIR uses timbral and beat information, which can be calculated using automatic computer audition techniques. In this paper, Pitch Histograms are defined and proposed as a way to represent the pitch content of music signals both in symbolic and audio form. This representation is evaluated in the context of automatic musical genre classification. A multiple-pitch detection algorithm for polyphonic signals is used to calculate Pitch Histograms for audio signals. In order to evaluate the extent and significance of errors resulting from the automatic multiple-pitch detection, automatic musical genre classification results from symbolic and audio data are compared. The comparison indicates that Pitch Histograms provide valuable information for musical genre classification. The results obtained for both symbolic and audio cases indicate that although pitch errors degrade classification performance for the audio case, Pitch Histograms can be effectively used for classification in both cases.

Improved phase vocoder time-scale modification of audio

The phase vocoder is a well established tool for time scaling and pitch shifting speech and audio signals via modification of their short-time Fourier transforms (STFTs). In contrast to time-domain time-scaling and pitch-shifting techniques, the phase vocoder is generally considered to yield high quality results, especially for large modification factors and/or polyphonic signals. However, the phase vocoder is also known for introducing a characteristic perceptual artifact, often described as "phasiness", "reverberation", or "loss of presence". This paper examines the problem of phasiness in the context of time-scale modification and provides new insights into its causes. Two extensions to the standard phase vocoder algorithm are introduced, and the resulting sound quality is shown to be significantly improved. Moreover, the modified phase vocoder is shown to provide a factor-of-two decrease in computational cost.

A novel approach for identifying polyphonic piano signals

The complexity of polyphonic sound signals and octave errors are the two main difficulties encountered in the identification of piano sounds [Chafe and Jaffe, Proc. ICASSP, Tokyo, 1289–1292 (1986)] [E. R. S. Pearson, Ph.D. thesis, University of Warwick (1991)]. Inharmonicity, which can be important in the case of the piano, is also often an obstacle to the identification of the fundamental frequency. The aim of this paper is to present a spectral identification method which takes advantage of inharmonicity to distinguish notes and avoid octave errors. Two independent steps are required to identify piano notes with this method. First, the distribution of the partials of each note must be determined, and a database containing this information must be, once and for all, created. Then, the notes are identified by comparing the incoming spectrum with the database. Frequential fluctuations on the positions of the partials are allowed during the identification procedure. This method identifies the notes of octaves C1 and B5 and treats monophonic as well as polyphonic signals (two, three, or even four notes played with a comparable hammer velocity).

A novel method for sorting piano note partials

A search procedure which takes into account the possible frequency fluctuations of the partials of piano notes is presented in this paper. This technique treats polyphonic sounds (as many notes as desired, provided that a sufficient number of their partials is present in the spectrum considered), in the same way as monophonic sounds and the computational time is linearly dependent on the number of partials recorded in a database. The identification of piano signals is difficult because of the problem of processing polyphonic signals. Depending on the model chosen, a note can be considered as harmonic [J. C. Brown, J. Acoust. Soc. Am. 92, 1394–1402 (1992)] or nonharmonic [Chafe and Jaffe, Proc. ICASSP, Tokyo, 1289–1292 (1986)] and the identification procedure is directly dependent on this model. The new sort procedure developed by the authors allows the sorting of harmonic, nonharmonic, or missing partials of a piano note.