Automatic Music Transcription Research Articles

DAFE-MSGAT: Dual-Attention Feature Extraction and Multi-Scale Graph Attention Network for Polyphonic Piano Transcription

Automatic music transcription (AMT) aims to convert raw audio signals into symbolic music. This is a highly challenging task in the fields of signal processing and artificial intelligence, and it holds significant application value in music information retrieval (MIR). Existing methods based on convolutional neural networks (CNNs) often fall short in capturing the time-frequency characteristics of audio signals and tend to overlook the interdependencies between notes when processing polyphonic piano with multiple simultaneous notes. To address these issues, we propose a dual attention feature extraction and multi-scale graph attention network (DAFE-MSGAT). Specifically, we design a dual attention feature extraction module (DAFE) to enhance the frequency and time-domain features of the audio signal, and we utilize a long short-term memory network (LSTM) to capture the temporal features within the audio signal. We introduce a multi-scale graph attention network (MSGAT), which leverages the various implicit relationships between notes to enhance the interaction between different notes. Experimental results demonstrate that our model achieves high accuracy in detecting the onset and offset of notes on public datasets. In both frame-level and note-level metrics, DAFE-MSGAT achieves performance comparable to the state-of-the-art methods, showcasing exceptional transcription capabilities.

END-TO-END AUTOMATIC MUSIC TRANSCRIPTION OF POLYPHONIC QANUN AND OUD MUSIC USING DEEP NEURAL NETWORK

This paper introduces an automatic music transcription model using Deep Neural Networks (DNNs), focusing on simulating the "trained ear" in music. It advances the field of signal processing and music technology, particularly in multi-instrument transcription involving traditional Turkish instruments, Qanun and Oud. Those instruments have unique timbral characteristics with early decay periods. The study involves generating basic combinations of multi-pitch datasets, training the DNN model on this data, and demonstrating its effectiveness in transcribing two-part compositions with high accuracy and F1 measures. The model's training involves understanding the fundamental characteristics of individual instruments, enabling it to identify and isolate complex patterns in mixed compositions. The primary goal is to empower the model to distinguish and analyze individual musical components, thereby enhancing applications in music production, audio engineering, and education

Jazz Trio Database: Automated Annotation of Jazz Piano Trio Recordings Processed Using Audio Source Separation

Recent advances in automatic music transcription have facilitated the creation of large databases of symbolic transcriptions of improvised music forms including jazz, where traditional notated scores are not normally available. In conjunction with music source separation models that enable audio to be “demixed” into separate signals for multiple instrument classes, these algorithms can also be applied to generate annotations for every musician in a performance. This can enable the analysis of interesting performer-level and ensemble-level features that have often been difficult to explore. To this end, we introduce Jazz Trio Database (JTD), a dataset of 44.5 h of jazz piano solos accompanied by bass and drums, with automatically generated annotations for each performer. These annotations consist of onset, beat, and downbeat timestamps, alongside MIDI for the piano soloist. Suitable recordings, broadly representative of the “straight-ahead” jazz style, were identified by scraping user-based listening and discographic data; source separation models were applied to isolate audio for each performer in the trio; annotations were generated by applying appropriate algorithms to both the separated and the mixed audio sources. Onset annotations generated by the pipeline achieved a mean F-measure of 0.94 when compared with ground truth annotations. We conduct several analyses of JTD, including with relation to swing and inter-performer synchronization. We anticipate that JTD will be useful in a variety of music information–retrieval tasks, including artist identification and expressive performance modeling. We have made JTD, including the annotations and associated source code, available at https://github.com/HuwCheston/Jazz-Trio-Database

Advanced emotion analysis: harnessing facial image processing and speech recognition through deep learning

The human face hides many secrets and is one of the most expressive human features. Human faces even contain hidden information about a person's personality. Considering the fundamental role of the human face, it is necessary to prepare appropriate deep-learning solutions that analyze human face data. This technology is becoming increasingly common in many industries, such as online retail, advertising testing, virtual makeovers, etc. For example, facial analysis technology now allows online shoppers to virtually apply makeup and try on jewelry or new glasses to get an accurate picture of what these products will look like. The human sense of hearing is a treasure trove of information about the current environment and the location and properties of sound-producing objects. For instance, we effortlessly absorb the sounds of birds singing outside the window, traffic passing in the distance, or the lyrics of a song on the radio. The human auditory system can process the intricate mix of sounds reaching our ears and create high-level abstractions of the environment by analyzing and grouping measured sensory signals. The process of obtaining segregation and identifying sources of a received complex acoustic signal, known as sound scene analysis, is a domain where the power of deep learning shines. The machine implementation of this functionality (separation and classification of sound sources) is pivotal in applications such as speech recognition in noise, automatic music transcription, searching and retrieving multimedia data, or recognizing emotions in statements.

Automatic Lyric Transcription and Automatic Music Transcription from Multimodal Singing

Automatic lyric transcription (ALT) refers to transcribing singing voices into lyrics, while automatic music transcription (AMT) refers to transcribing singing voices into note events, i.e., musical MIDI notes. Despite these two tasks having significant potential for practical application, they are still nascent. This is because the transcription of lyrics and note events solely from singing audio is notoriously difficult due to the presence of noise contamination, e.g., musical accompaniment, resulting in a degradation of both the intelligibility of sung lyrics and the recognizability of sung notes. To address this challenge, we propose a general framework for implementing multimodal ALT and AMT systems. Additionally, we curate the first multimodal singing dataset, comprising N20EMv1 and N20EMv2, which encompasses audio recordings and videos of lip movements, together with ground truth for lyrics and note events. For model construction, we propose adapting self-supervised learning models from the speech domain as acoustic encoders and visual encoders to alleviate the scarcity of labeled data. We also introduce a residual cross-attention mechanism to effectively integrate features from the audio and video modalities. Through extensive experiments, we demonstrate that our single-modal systems exhibit state-of-the-art performance on both ALT and AMT tasks. Subsequently, through single-modal experiments, we also explore the individual contributions of each modality to the multimodal system. Finally, we combine these and demonstrate the effectiveness of our proposed multimodal systems, particularly in terms of their noise robustness.

IRAWNET: A Method for Transcribing Indonesian Classical Music Notes Directly from Multichannel Raw Audio

A challenging task when developing real-time Automatic Music Transcription (AMT) methods is directly leveraging inputs from multichannel raw audio without any handcrafted signal transformation and feature extraction steps. The crucial problems are that raw audio only contains an amplitude in each timestamp, and the signals of the left and right channels have different amplitude intensities and onset times. Thus, this study addressed these issues by proposing the IRawNet method with fused feature layers to merge different amplitude from multichannel raw audio. IRawNet aims to transcribe Indonesian classical music notes. It was validated with the Gamelan music dataset. The Synthetic Minority Oversampling Technique (SMOTE) overcame the class imbalance of the Gamelan music dataset. Under various experimental scenarios, the performance effects of oversampled data, hyperparameters tuning, and fused feature layers are analyzed. Furthermore, the performance of the proposed method was compared with Temporal Convolutional Network (TCN), Deep WaveNet, and the monochannel IRawNet. The results proved that proposed method almost achieves superior results in entire metric performances with 0.871 of accuracy, 0.988 of AUC, 0.927 of precision, 0.896 of recall, and 0.896 of F1 score.

Teaching Integration of Piano and Traditional Music Elements in Colleges and Universities Based on Network Flow Optimization

Abstract In this paper, from the basis of piano music theory, pitch and twelve equal temperaments are studied, and due to the characteristics of the piano’s construction, there are different timbres and harmonic structures. In order to realize the digitization of music signals for piano teaching, the collected analog signals are converted into digital signals by Fourier transform, the samples are inputted into the neural network model to get the output results, and the loss function based on Markov’s music language model in piano teaching is used to find the partial derivatives of the parameters to realize the maximization of the efficiency of the piano pitch and note recognition. According to the multi-task learning in neural networks and the envelope curve of notes, we construct the automatic music transcription model in piano teaching based on CNN-HMM and use the simulation experiment analysis to empirically analyze the piano teaching based on CNNHMM. The results show that the dual-channel audio results are improved by 2.68% in F1 value, 3.18% in accuracy and 2.61% in recall than the mono audio results. That is, dual-channel audio has richer information than mono audio, which enables the CNN-HNN network to learn richer features and thus improve the pitch and note recognition results. In this paper, the reliability of the teaching quality evaluation results of the CNN-HMM-based quantitative assessment model for facilitating the combination of electronic organ and piano teaching concepts ranges from 78.1% to 85.4%. This study provides guiding value for the development of digital piano teaching, which is of great significance for the innovative reform of piano teaching in colleges and universities.

Note-level singing melody transcription with transformers

Recognizing a singing melody from an audio signal in terms of the music notes’ pitch onset and offset, referred to as note-level singing melody transcription, has been studied as a critical task in the field of automatic music transcription. The task is challenging due to the different timbre and vibrato of each vocal and the ambiguity of onset and offset of the human voice compared with other instrumental sounds. This paper proposes a note-level singing melody transcription model using sequence-to-sequence Transformers. The singing melody annotation is expressed as a monophonic melody sequence and used as a decoder sequence. Overlapping decoding is introduced to solve the problem of the context between segments being broken. Applying pitch augmentation and and adding noisy dataset with data cleansing turns out to be effective in preventing overfitting and generalizing the model performance. Ablation studies demonstrate the effects of the proposed techniques in note-level singing melody transcription, both quantitatively and qualitatively. The proposed model outperforms other models in note-level singing melody transcription performance for all the metrics considered. For fundamental frequency metrics, the voice detection performance of the proposed model is comparable to that of a vocal melody extraction model. Finally, subjective human evaluation demonstrates that the results of the proposed models are perceived as more accurate than the results of a previous study.

Harmonizing minds and machines: survey on transformative power of machine learning in music.

This survey explores the symbiotic relationship between Machine Learning (ML) and music, focusing on the transformative role of Artificial Intelligence (AI) in the musical sphere. Beginning with a historical contextualization of the intertwined trajectories of music and technology, the paper discusses the progressive use of ML in music analysis and creation. Emphasis is placed on present applications and future potential. A detailed examination of music information retrieval, automatic music transcription, music recommendation, and algorithmic composition presents state-of-the-art algorithms and their respective functionalities. The paper underscores recent advancements, including ML-assisted music production and emotion-driven music generation. The survey concludes with a prospective contemplation of future directions of ML within music, highlighting the ongoing growth, novel applications, and anticipation of deeper integration of ML across musical domains. This comprehensive study asserts the profound potential of ML to revolutionize the musical landscape and encourages further exploration and advancement in this emerging interdisciplinary field.

High-Quality and Reproducible Automatic Drum Transcription from Crowdsourced Data

Within the broad problem known as automatic music transcription, we considered the specific task of automatic drum transcription (ADT). This is a complex task that has recently shown significant advances thanks to deep learning (DL) techniques. Most notably, massive amounts of labeled data obtained from crowds of annotators have made it possible to implement large-scale supervised learning architectures for ADT. In this study, we explored the untapped potential of these new datasets by addressing three key points: First, we reviewed recent trends in DL architectures and focused on two techniques, self-attention mechanisms and tatum-synchronous convolutions. Then, to mitigate the noise and bias that are inherent in crowdsourced data, we extended the training data with additional annotations. Finally, to quantify the potential of the data, we compared many training scenarios by combining up to six different datasets, including zero-shot evaluations. Our findings revealed that crowdsourced datasets outperform previously utilized datasets, and regardless of the DL architecture employed, they are sufficient in size and quality to train accurate models. By fully exploiting this data source, our models produced high-quality drum transcriptions, achieving state-of-the-art results. Thanks to this accuracy, our work can be more successfully used by musicians (e.g., to learn new musical pieces by reading, or to convert their performances to MIDI) and researchers in music information retrieval (e.g., to retrieve information from the notes instead of audio, such as the rhythm or structure of a piece).

Piano automatic transcription based on transformer

Recent years, research on automatic music transcription has made significant progress as deep learning techniques have been validated to demonstrate strong performance in complex data applications. Although the existing work is exciting, they all rely on specific domain knowledge to enable the design of model architectures and training modes for different tasks. At the same time, the noise generated in the process of automatic music transcription data collection cannot be ignored, which makes the existing work unsatisfactory. To address the issues highlighted above, we propose an end-to-end framework based on Transformer. Through the encoder-decoder structure, we realize the direct conversion of the spectrogram of the collected piano audio to MIDI output. Further, to remove the impression of environmental noise on transcription quality, we design a training mechanism mixed with white noise to improve the robustness of our proposed model. Our experiments on the classic piano transcription datasets show that the proposed method can greatly improve the quality of automatic music transcription.

Comparative Analysis of Deep Learning Architectures and Vision Transformers for Musical Key Estimation

The musical key serves as a crucial element in a piece, offering vital insights into the tonal center, harmonic structure, and chord progressions while enabling tasks such as transposition and arrangement. Moreover, accurate key estimation finds practical applications in music recommendation systems and automatic music transcription, making it relevant across academic and industrial domains. This paper presents a comprehensive comparison between standard deep learning architectures and emerging vision transformers, leveraging their success in various domains. We evaluate their performance on a specific subset of the GTZAN dataset, analyzing six different deep learning models. Our results demonstrate that DenseNet, a conventional deep learning architecture, achieves remarkable accuracy of 91.64%, outperforming vision transformers. However, we delve deeper into the analysis to shed light on the temporal characteristics of each deep learning model. Notably, the vision transformer and SWIN transformer exhibit a slight decrease in overall performance (1.82% and 2.29%, respectively), yet they demonstrate superior performance in temporal metrics compared to the DenseNet architecture. The significance of our findings lies in their contribution to the field of musical key estimation, where accurate and efficient algorithms play a pivotal role. By examining the strengths and weaknesses of deep learning architectures and vision transformers, we can gain valuable insights for practical implementations, particularly in music recommendation systems and automatic music transcription. Our research provides a foundation for future advancements and encourages further exploration in this area.

Polyphonic piano transcription based on graph convolutional network

The automatic music transcription (AMT) task is designed to convert raw performance audio signals into digital representations of symbolic music for possible computational musicology. In polyphonic music, there are multiple notes and they may appear at the same time. The combined output of multiple notes faces the problem of dimension explosion, which makes it difficult to achieve accurate transcription. To overcome the above challenge, a deep learning model based on graph convolution network (CR-GCN) is proposed to cope with the problem of dimension explosion by exploring the interdependence between musical notes. The model is divided into two parts, i.e., feature learning and label learning. Feature learning is composed of serial convolutional neural network (CNN) and recurrent neural network (RNN), which aims to extract temporal and spatial features from the input music signals. Label learning consists of graph convolutional network (GCN), which is adopted to model the interdependence between notes. The CR-GCN model can be end-to-end trainable through joint training feature networks and label networks. Experiments on public polyphonic music data sets show that the proposed method is able to mine more co-existing notes, and it is superior to existing methods in both frame-level and note-level indexes. Moreover, visual analysis shows that the learned interdependence between notes has good explainability in music theory.

Creating a new research community on detection and classification of acoustic scenes and events: Lessons from the first ten years of DCASE challenges and workshops

Research work on automatic speech recognition and automatic music transcription has been around for several decades, supported by dedicated conferences or conference sessions. However, while individual researchers have been working on recognition of more general environmental sounds, until ten years ago there were no regular workshops or conference sessions where this research, or its researchers, could be found. There was also little available data for researchers to work on or to benchmark their work. In this talk we will outline how a new research community working on Detection and Classification of Acoustic Scenes and Events (DCASE) has grown over the last ten years, from two challenges on acoustic scene classification and sound event detection with a small workshop poster session, to an annual data challenge with six tasks and a dedicated annual workshop, attracting hundreds of delegates and strong industry interest. We will also describe how the analysis methods have evolved, from mel frequency cepstral coefficients (MFCCs) or cochelograms classified by support vector machines (SVMs) or hidden Markov models (HMMs), to deep learning methods such as transfer learning, transformers, and self-supervised learning. We will finish by suggesting some potential future directions for automatic sound recognition and the DCASE community.

Late multimodal fusion for image and audio music transcription

Music transcription, which deals with the conversion of music sources into a structured digital format, is a key problem for Music Information Retrieval (MIR). When addressing this challenge in computational terms, the MIR community follows two lines of research: music documents, which is the case of Optical Music Recognition (OMR), or audio recordings, which is the case of Automatic Music Transcription (AMT). The different nature of the aforementioned input data has conditioned these fields to develop modality-specific frameworks. However, their recent definition in terms of sequence labeling tasks leads to a common output representation, which enables research on a combined paradigm. In this respect, multimodal image and audio music transcription comprises the challenge of effectively combining the information conveyed by image and audio modalities. In this work, we explore this question at a late-fusion level: we study four combination approaches in order to merge, for the first time, the hypotheses regarding end-to-end OMR and AMT systems in a lattice-based search space. The results obtained for a series of performance scenarios–in which the corresponding single-modality models yield different error rates–showed interesting benefits of these approaches. In addition, two of the four strategies considered significantly improve the corresponding unimodal standard recognition frameworks.

A Comprehensive Review on Audio based Musical Instrument Recognition: Human-Machine Interaction Towards Industry 4.0

Over the last two decades, the application of machine technology has shifted from industrial to residential use. Further, advances in hardware and software sectors have led machine technology to its utmost application, the human-machine interaction, a multimodal communication. Multimodal communication refers to the integration of various modalities of information like speech, image, music, gesture, and facial expressions. Music is the non-verbal type of communication that humans often use to express their minds. Thus, Music Information Retrieval (MIR) has become a booming field of research and has gained a lot of interest from the academic community, music industry, and vast multimedia users. The problem in MIR is accessing and retrieving a specific type of music as demanded from the extensive music data. The most inherent problem in MIR is music classification. The essential MIR tasks are artist identification, genre classification, mood classification, music annotation, and instrument recognition. Among these, instrument recognition is a vital sub-task in MIR for various reasons, including retrieval of music information, sound source separation, and automatic music transcription. In recent past years, many researchers have reported different machine learning techniques for musical instrument recognition and proved some of them to be good ones. This article provides a systematic, comprehensive review of the advanced machine learning techniques used for musical instrument recognition. We have stressed on different audio feature descriptors of common choices of classifier learning used for musical instrument recognition. This review article emphasizes on the recent developments in music classification techniques and discusses a few associated future research problems.

Automatic music transcription for sitar music analysis

The automatic transcription for non-Western music presents unique challenges, particularly because the result is often imperfect. How to improve automatic music transcription (AMT) for computational musicology research are open questions. In this paper, we report a novel AMT method and a set of analysis strategies for sitar recordings. With a detailed introduction of the proposed harmonic multi-layered cepstrum (HMLC) and combined frequency-periodicity (CFP) signal representations, we also provide the tonal, temporal and structural analysis based on the AMT result. Our proposed AMT method produces improved results, and we further demonstrate the advantages of using this AMT method in musicological analysis.

AI System Design for Robotic Hand to Play the Piano

Robotic and Artificial Intelligent (AI) have been introduced as a key factor for industry revolution4.0. Many industries, such as manufacturing, agriculture, logistics, supply chain, and so on, are transformed and applied robotic and AI to enhance productivity and reduce cost. AI in creative work is very challenging, especially in music. This paper presents a system to enable the robotic arm to play piano notes with minimal errors. We used the knowledge of Optical music recognition (OMR), Automatic music transcription (AMT), Music source separation (MSS), and the elimination of robot arm cycle times problems for creating this system. The robotic arm used in testing with this system was LEGO. It can perform 4 functions. The first function is to play piano notes from sheet piano using Sheet Vision and Tesseract-OCR. Note reading accuracy is 21.36%, and note reading accuracy with note duration is 13.46%. The second function is to play the piano like a piano sound separate from a music file using Spleeter and Onsets and Frames. Piano note accuracy is 58.32%, piano note onset time error is ± 0.17, and piano note duration time error is ± 0.65. The third function is to play piano notes from real-time piano sounds using Onsets and Frames Real-time mode. The last function is to play the piano notes from the brain waves by comparing the frequency of the brain waves with the frequency of the piano notes. The design and experimental results are explained in this paper.

Transfer Learning for Audio Waveform to Guitar Chord Spectrograms Using the Convolution Neural Network

Automatic chord recognition has always been approached as a broad music audition task. The desired output is a succession of time-aligned discrete chord symbols, such as GMaj and Asus2. Automatic music transcription is the process of converting a musical recording into a human-readable and interpretable representation. When dealing with polyphonic sounds or removing certain limits, automatic music transcription remains a difficult undertaking. A guitar, for example, presents a greater challenge, as guitarists can play the same note in a variety of places. The study makes use of CNN functionality to generate the guitar tab; initially, the constant-Q transform was used to turn the input audio file into short time spectrograms that the CNN model utilises to analyse the chord. The paper developed a method for extracting chord sequences and notes from audio recordings of solo guitar performances. For intervals in the supplied audio, the proposed approach outputs chord names and fret-board notes. The model described here has been refined to achieve an accuracy of 88.7%. The model’s ability to properly tag audio clips is an incredible advancement.

Automatic Music Transcription Using Fourier Transform for Monophonic and Polyphonic Audio File

Musical sheet is an important tool for musicians that enables musicians to communicate with each other and help musicians to learn a composition of a song. Sometimes, musicians face an obstacle when they cannot find the musical sheet to learn a new song or it may require payment to get the sheet. The solution for this problem is to learn the song by figuring out the composition of a song using music transcription. Music Transcription is the process of music information retrieval to produce musical notation. Music Transcription using a computational method often called Automatic Music Transcription by upload the audio file as an input and generate musical sheet. The proposed method is solved using Note Value Detection to separate windows by the detected note values and Fourier Transform to recognize the frequency from each window. This study is evaluating the system using three variables; note value, pitch accuracy, and extra notes. The study shows that note value and pitch detection gives a relatively small percentage error. Meanwhile, extra note detection gives a relatively moderate percentage error in every polyphonic file.