Automatic Music Transcription System Research Articles

Automatic Lyric Transcription and Automatic Music Transcription from Multimodal Singing

Automatic lyric transcription (ALT) refers to transcribing singing voices into lyrics, while automatic music transcription (AMT) refers to transcribing singing voices into note events, i.e., musical MIDI notes. Despite these two tasks having significant potential for practical application, they are still nascent. This is because the transcription of lyrics and note events solely from singing audio is notoriously difficult due to the presence of noise contamination, e.g., musical accompaniment, resulting in a degradation of both the intelligibility of sung lyrics and the recognizability of sung notes. To address this challenge, we propose a general framework for implementing multimodal ALT and AMT systems. Additionally, we curate the first multimodal singing dataset, comprising N20EMv1 and N20EMv2, which encompasses audio recordings and videos of lip movements, together with ground truth for lyrics and note events. For model construction, we propose adapting self-supervised learning models from the speech domain as acoustic encoders and visual encoders to alleviate the scarcity of labeled data. We also introduce a residual cross-attention mechanism to effectively integrate features from the audio and video modalities. Through extensive experiments, we demonstrate that our single-modal systems exhibit state-of-the-art performance on both ALT and AMT tasks. Subsequently, through single-modal experiments, we also explore the individual contributions of each modality to the multimodal system. Finally, we combine these and demonstrate the effectiveness of our proposed multimodal systems, particularly in terms of their noise robustness.

Late multimodal fusion for image and audio music transcription

Music transcription, which deals with the conversion of music sources into a structured digital format, is a key problem for Music Information Retrieval (MIR). When addressing this challenge in computational terms, the MIR community follows two lines of research: music documents, which is the case of Optical Music Recognition (OMR), or audio recordings, which is the case of Automatic Music Transcription (AMT). The different nature of the aforementioned input data has conditioned these fields to develop modality-specific frameworks. However, their recent definition in terms of sequence labeling tasks leads to a common output representation, which enables research on a combined paradigm. In this respect, multimodal image and audio music transcription comprises the challenge of effectively combining the information conveyed by image and audio modalities. In this work, we explore this question at a late-fusion level: we study four combination approaches in order to merge, for the first time, the hypotheses regarding end-to-end OMR and AMT systems in a lattice-based search space. The results obtained for a series of performance scenarios–in which the corresponding single-modality models yield different error rates–showed interesting benefits of these approaches. In addition, two of the four strategies considered significantly improve the corresponding unimodal standard recognition frameworks.

A perceptual measure for evaluating the resynthesis of automatic music transcriptions

This study focuses on the perception of music performances when contextual factors, such as room acoustics and instrument, change. We propose to distinguish the concept of “performance” from the one of “interpretation”, which expresses the “artistic intention”. Towards assessing this distinction, we carried out an experimental evaluation where 91 subjects were invited to listen to various audio recordings created by resynthesizing MIDI data obtained through Automatic Music Transcription (AMT) systems and a sensorized acoustic piano. During the resynthesis, we simulated different contexts and asked listeners to evaluate how much the interpretation changes when the context changes. Results show that: (1) MIDI format alone is not able to completely grasp the artistic intention of a music performance; (2) usual objective evaluation measures based on MIDI data present low correlations with the average subjective evaluation. To bridge this gap, we propose a novel measure which is meaningfully correlated with the outcome of the tests. In addition, we investigate multimodal machine learning by providing a new score-informed AMT method and propose an approximation algorithm for the p-dispersion problem.

Multimodal image and audio music transcription

Optical Music Recognition (OMR) and Automatic Music Transcription (AMT) stand for the research fields that aim at obtaining a structured digital representation from sheet music images and acoustic recordings, respectively. While these fields have traditionally evolved independently, the fact that both tasks may share the same output representation poses the question of whether they could be combined in a synergistic manner to exploit the individual transcription advantages depicted by each modality. To evaluate this hypothesis, this paper presents a multimodal framework that combines the predictions from two neural end-to-end OMR and AMT systems by considering a local alignment approach. We assess several experimental scenarios with monophonic music pieces to evaluate our approach under different conditions of the individual transcription systems. In general, the multimodal framework clearly outperforms the single recognition modalities, attaining a relative improvement close to 40% in the best case. Our initial premise is, therefore, validated, thus opening avenues for further research in multimodal OMR-AMT transcription.

Humanities and engineering perspectives on music transcription

Abstract Music transcription is a process of creating a notation of musical sounds. It has been used as a basis for the analysis of music from a wide variety of cultures. Recent decades have seen an increasing amount of engineering research within the field of Music Information Retrieval that aims at automatically obtaining music transcriptions in Western staff notation. However, such approaches are not widely applied in research in ethnomusicology. This article aims to bridge interdisciplinary gaps by identifying aspects of proximity and divergence between the two fields. As part of our study, we collected manual transcriptions of traditional dance tune recordings by eighteen transcribers. Our method employs a combination of expert and computational evaluation of these transcriptions. This enables us to investigate the limitations of automatic music transcription (AMT) methods and computational transcription metrics that have been proposed for their evaluation. Based on these findings, we discuss promising avenues to make AMT more useful for studies in the Humanities. These are, first, assessing the quality of a transcription based on an analytic purpose; secondly, developing AMT approaches that are able to learn conventions concerning the transcription of a specific style; thirdly, a focus on novice transcribers as users of AMT systems; and, finally, considering target notation systems different from Western staff notation.

High-Resolution Piano Transcription With Pedals by Regressing Onset and Offset Times

Automatic music transcription (AMT) is the task of transcribing audio recordings into symbolic representations. Recently, neural network based methods have been applied to AMT, and have achieved state-of-the-art results. However, many previous systems only detect onset and offset of notes in frame-wise, so the transcription resolution is limited to the frame hop size. There is a lack of research of using different strategies to encode onset and offset targets for training. In addition, previous AMT systems are sensitive to the misaligned onset and offset labels of audio recordings. Furthermore, there are limited research of sustain pedal transcription on large-scale datasets. In this article, we propose a high-resolution AMT system trained by regressing precise onset and offset times of piano notes. At inference, we propose an algorithm to analytically calculate the precise onset and offset times of piano notes and pedal events. We show that our AMT system is robust to misaligned onset and offset labels compared to previous systems. Our proposed system achieves an onset F1 of 96.72% on the MAESTRO dataset, outperforming previous onsets and frames system of 94.80%. Our system achieves a pedal onset F1 score of 91.86%, which is the first benchmark result on the MAESTRO dataset. We have released the source code and checkpoints of our work at https://github.com/bytedance/piano_transcription.

Investigating the Perceptual Validity of Evaluation Metrics for Automatic Piano Music Transcription

Automatic Music Transcription (AMT) is usually evaluated using low-level criteria, typically by counting the number of errors, with equal weighting. Yet, some errors (e.g. out-of-key notes) are more salient than others. In this study, we design an online listening test to gather judgements about AMT quality. These judgements take the form of pairwise comparisons of transcriptions of the same music by pairs of different AMT systems. We investigate how these judgements correlate with benchmark metrics, and find that although they match in many cases, agreement drops when comparing pairs with similar scores, or pairs of poor transcriptions. We show that onset-only notewise F-measure is the benchmark metric that correlates best with human judgement, all the more so with higher onset tolerance thresholds. We define a set of features related to various musical attributes, and use them to design a new metric that correlates significantly better with listeners’ quality judgements. We examine which musical aspects were important to raters by conducting an ablation study on the defined metric, highlighting the importance of the rhythmic dimension (tempo, meter). We make the collected data entirely available for further study, in particular to evaluate the perceptual relevance of new AMT metrics.

Estimating the Rank of a Nonnegative Matrix Factorization Model for Automatic Music Transcription Based on Stein’s Unbiased Risk Estimator

In this paper, methods to estimate the number of basis vectors of the nonnegative matrix factorization (NMF) of automatic music transcription (AMT) systems are proposed. Previously, studies on NMF-based AMT have demonstrated that the number of basis vectors affects the performance and that the number of note events can be a good selection as the rank of NMF. However, many NMF-based AMT methods do not provide a method to estimate the appropriate number of basis vectors; instead, the number is assumed to be given in advance, even though the number of basis vectors significantly affects the algorithm’s performance. Recently, based on Bayesian methods, certain estimation algorithms for the number of basis vectors have been proposed; however, they are not designed to be used as music transcription algorithms but are components of specific NMF methods and thus cannot be used generally as NMF-based transcription algorithms. Our proposed estimation algorithms are based on eigenvalue decomposition and Stein’s unbiased risk estimator (SURE). Because the SURE method requires variance in undesired components as a priori knowledge, the proposed algorithms estimate the value using random matrix theory and first and second onset information in the input music signal. Experiments were then conducted for the AMT task using the MIDI-aligned piano sounds (MAPS) database, and these algorithms were compared with variational NMF, gamma process NMF, and NMF with automatic relevance determination algorithms. Based on experimental results, the conventional NMF-based transcription algorithm with the proposed rank estimation algorithms demonstrated enhanced F1 score performances of 2–3% compared to the algorithms. While the performance advantages are not significantly large, the results are meaningful because the proposed algorithms are lightweight, are easy to combine with any other NMF methods that require an a priori rank parameter, and do not have setting parameters that considerably affect the performance.

Automatic music transcription for traditional woodwind instruments sopele

Sopela is a traditional hand-made woodwind instrument, commonly played in pair, characteristic to the Istrian peninsula in western Croatia. Its piercing sound, accompanied by two-part singing in the hexatonic Istrian scale, is registered in the UNESCO Representative List of the Intangible Cultural Heritage of Humanity. This paper presents an insight study of automatic music transcription (AMT) for sopele tunes. The process of converting audio inputs into human-readable musical scores involves multi-pitch detection and note tracking. The proposed solution supports this process by utilising frequency-feature extraction, supervised machine learning (ML) algorithms, and postprocessing heuristics. We determined the most favourable tone-predicting model by applying grid search for two state-of-the-art ML techniques, optionally coupled with frequency-feature extraction. The model achieved promising transcription accuracy for both monophonic and polyphonic music sources encompassed in the originally developed dataset. In addition, we developed a proof-of-concept AMT system, comprised of a client mobile application and a server-side API. While the mobile application records, tags and uploads audio sources, the back-end server applies the presented procedure for converting recorded music into a common notation to be delivered as a transcription result. We thus demonstrate how collecting and preserving traditional sopele music, performed in real-life occasions, can be effortlessly accomplished on-the-go.

Instrument Learning and Sparse NMD for Automatic Polyphonic Music Transcription

In this paper, an automatic music transcription (AMT) algorithm based on a supervised non-negative matrix decomposition (NMD) is discussed. In particular, a novel approach for enhancing the sparsity of the solution is proposed. It consists of a two-step processing in which the NMD is solved joining a $\ell _2$ regularization and a threshold filtering. In the first step, the NMD is performed with the $\ell _2$ regularization in order to get an overall selection of the notes most likely appearing in the monotimbral musical excerpt. In the second step, a threshold filtering followed by another $\ell _2$ regularized NMD are repeatedly performed in order to progressively reduce the dictionary matrix and to refine the notes transcription. Furthermore, a user-oriented instrument learning procedure has been conceived and proposed. The proposed AMT system has been tested upon the dataset collected by the LabROSA laboratories considering the transcription of three different pianos. Moreover, it has been validated through a comparison with a regularized NMD and with three open source AMT software. The results prove the effectiveness of the proposed two-step processing in enhancing the sparsity of the solution and in improving the transcription accuracy. Moreover, the proposed system shows promising performance in both multi-F0 and note tracking tasks, obtaining better transcription accuracy than the competing algorithms in most tests.

An automatic music transcription system dedicated to the repertoires of the marovany zither

The meeting between Music information retrieval (MIR) and Ethnomusicology has given way to a new scientific discipline called Computational Ethnomusicology, which aims to adapt and develop MIR tools for the multiple corpus of ethnic music. In this paper, we are interested in developing an automatic transcription system dedicated to the marovany musical repertoires. The marovany repertoires are mostly orally transmitted, resulting from a process of memorization/transformation of original base musical motives. These motives represent an important culture patrimony, and are evolving continually under the influences of other musical practices and genres due to globalization. Our transcription system consists mainly of a Probabilistic Latent Component Analysis algorithm post-processed with Hidden Markov Model. It incorporates repertoire-specific music-related prior knowledge about timbre, music language and playing style. The impact of each knowledge component on transcription results has been comparatively evaluated. The most beneficial prior combination brought a transcription gain of 7.4% from the note-based F-measures of state-of-the-art transcription system performance.

Real-Time Piano Music Transcription Based on Computer Vision

One important problem in musical information retrieval is automatic music transcription, which is an automated conversion process from played music to a symbolic notation such as MIDI file. Since the accuracy of previous audio-based transcription systems is not satisfactory, we propose an innovative computer vision-based automatic music transcription system named claVision to perform piano music transcription. Instead of processing the music audio, the system performs the transcription only from the video performance captured by a camera mounted over the piano keyboard. In this paper, we describe the architecture and the algorithms used in claVision. The claVision system has a high accuracy ( ${F_1}$ score over 0.95) and a very low latency (about 7.0 ms) in real-time music transcription, even under different illumination conditions. This technology can also be used for other musical keyboard instruments.

An investigation of prior knowledge in Automatic Music Transcription systems.

Automatic transcription of music is a long-studied research field with many operational systems available commercially. In this paper, a generic transcription system able to host various prior knowledge parameters has been developed, followed by an in-depth investigation of their impact on music transcription. Explicit links between musical knowledge and algorithmic formalism have been made. Musical knowledge covers classes of timbre, musicology, and playing style of an instrument repertoire. An evaluation sound corpus gathering musical pieces played by human performers from three different instrument repertoires, namely, classical piano, steel-string acoustic guitar, and the marovany zither from Madagascar, has been developed. The different components of musical knowledge have been successively incorporated in a complete transcription system, consisting mainly of a Probabilistic Latent Component Analysis algorithm post-processed with a Hidden Markov Model, and their impact on transcription results have been comparatively evaluated.

A Novel System for Music Learning using Low Complexity Algorithms

This paper introduces a music learning system that uses new low complexity algorithms and aims to solve the four most common problems faced by self-learning beginner pianists: reading music sheets, playing fast tempo music pieces, verifying the key of a music piece, and finally evaluating their own performances. In order to achieve these aims, the system proposes a monophonic automatic music transcription system capable of detecting notes in the range from G2 to G6. It uses an autocorrelation algorithm along with a binary search based algorithm in order to map the detected frequencies of the individual notes of a musical piece to the nearest musical frequencies. To enable playing fast music, the system uses a MIDI player equipped with a virtual piano as well as section looping and speed manipulation functionalities to enable the user to start learning a musical piece slowly and build up speed. Furthermore, it applies the Krumhansl-Schmuckler key-finding algorithm along with the correlation algorithm to identify the key of a musical piece. A musical performance evaluation algorithm is also introduced which compares the original performance with that of the learner’s producing a quantitative similarity measure between the two. The experimental evaluation shows that the system is capable of detecting notes in the range from G2 to G6 with an accuracy of 88.7% in addition to identifying the key of a musical piece with an accuracy of 97.1%.

A Compositional Automatic Music Transcription System for Computersynthesized Music

Automatic music transcription can be widely used for content-based music retrieval, low rate coding and automatic musical accompaniment system and so on. Multiple fundamental frequency estimation, or Multiple-F0 estimation, is one of the most important problems in automatic music transcription, but it has not been well resolved up to now. This paper presents a machine learning methods using harmonic matching and iterative deletion for computer-synthesized music specifically to Multiple-F0 estimation and builds an efficient automatic music transcription system. Computersynthesized music is almost free from similar instruments of the differences between different individuals, so it is a good research object. It can be shown in this paper that the experimental results indicate that this method has very good recognition results.

A Combined Mathematical Treatment for a Special Automatic Music Transcription System

This paper presents a combined mathematical treatment for a special automatic music transcription system. This system is specially made for computer‐synthesized music. The combined mathematical treatment includes harmonic selection, matrix analysis, and probability analysis method. The algorithm reduces dimension by PCA and selects candidates first by human auditory model and harmonic structures of notes. It changes the multiple‐F0 estimation question into a mathematical problem and solves it in a mathematical way. It can be shown in this paper that the experimental results indicate that this method has very good recognition results.

단일 음원 노래에서 음표의 조합 규칙을 이용한 마디 위치 찾기

본 논문은 한 마디 내에서 나타나는 음표들을 조합했을 때 규칙적인 배수관계가 있음을 이용하여 단음으로 구성된 노래에서 정확하게 마디 위치를 찾는 방법을 제안한다. 제안된 방법에서는 먼저 노래의 개별음의 길이들을 찾아내고, 이들 개별음 길이들의 배수관계를 적용한 규칙들을 이용한다. 이렇게 찾아진 마디정보는 자동 음악 채보 시스템의 사전 정보로 활용될 수 있는 박자 및 빠르기 등을 위한 원천적인 정보를 추출하는데 활용될 수 있다. 남성 및 여성의 단음으로 구성된 12곡의 노래들에 대한 실험 결과 1곡을 제외한 11곡 노래들의 마디위치를 정확히 검출 했으며, 이렇게 찾은 마디 정보와 음악적 이론을 이용하여 곡의 박자 및 빠르기 정보 등을 추출할 수 있다. There exist some regular multiple relations in the intervals of notes when they are combined within one measure. This paper presents a method to find the exact measure positions in monophonic song based on those relations. In the proposed method the individual intervals are segmented at first and the rules that state the multiple relations are used to find the measure position. The measures can be applied as the foundational information for extracting beat and tempo of a song which can be used as background knowledge of automatic music transcription system. The proposed method exactly detected the measure positions of 11 songs out of 12 songs except one song which consist of monophonic voice song of the men and women. Also one can extract the information of beat and tempo of a song using the information about extracted measure positions with music theory.

Method for automatically transcribing music and apparatus therefore

An automatic music transcription method and system for generating a muscial score from an input acoustic signal. The acoustic signal may include vocal songs, vocal humming, and music from musical instruments. The system comprises means for extracting pitch information and power information from the input acoustic signal, for correcting the pitch information based on the deviation of the acoustic signal relative to an absolute musical scale, for dividing the acoustic signal into a set of single-sound segments using the corrected pitch information, dividing the acoustic signal into a second set of single-sound segments this time using changes in the power information, for dividing the acoustic signal in still greater detail using information contained in both previous segmentations, for associating each segment with a musical interval of an absolute musical scale, and for determining single-sounds segments depending on whether or not the musical intervals of adjacent segments are identical, for determining the key of the acoustic signal, for correcting the placement of the segments on the musical scale of the determined key using the pitch information, for determining the time and tempo of the acoustic signal using this placement, and for compiling musical score data using the determined musical scale, sound length, key, time, and tempo of the acoustic signal.