Automatic Transcription Research Articles

Transfer Learning for Audio Waveform to Guitar Chord Spectrograms Using the Convolution Neural Network

Automatic chord recognition has always been approached as a broad music audition task. The desired output is a succession of time-aligned discrete chord symbols, such as GMaj and Asus2. Automatic music transcription is the process of converting a musical recording into a human-readable and interpretable representation. When dealing with polyphonic sounds or removing certain limits, automatic music transcription remains a difficult undertaking. A guitar, for example, presents a greater challenge, as guitarists can play the same note in a variety of places. The study makes use of CNN functionality to generate the guitar tab; initially, the constant-Q transform was used to turn the input audio file into short time spectrograms that the CNN model utilises to analyse the chord. The paper developed a method for extracting chord sequences and notes from audio recordings of solo guitar performances. For intervals in the supplied audio, the proposed approach outputs chord names and fret-board notes. The model described here has been refined to achieve an accuracy of 88.7%. The model’s ability to properly tag audio clips is an incredible advancement.

Automatic Music Transcription Using Fourier Transform for Monophonic and Polyphonic Audio File

Musical sheet is an important tool for musicians that enables musicians to communicate with each other and help musicians to learn a composition of a song. Sometimes, musicians face an obstacle when they cannot find the musical sheet to learn a new song or it may require payment to get the sheet. The solution for this problem is to learn the song by figuring out the composition of a song using music transcription. Music Transcription is the process of music information retrieval to produce musical notation. Music Transcription using a computational method often called Automatic Music Transcription by upload the audio file as an input and generate musical sheet. The proposed method is solved using Note Value Detection to separate windows by the detected note values and Fourier Transform to recognize the frequency from each window. This study is evaluating the system using three variables; note value, pitch accuracy, and extra notes. The study shows that note value and pitch detection gives a relatively small percentage error. Meanwhile, extra note detection gives a relatively moderate percentage error in every polyphonic file.

The Evolution of an Innovative Online Task to Monitor Children's Oral Narrative Development.

Oral narrative abilities are an important measure of children's language competency and have predictive value for children's later academic performance. Research and development underway in New Zealand is advancing an innovative online oral narrative task. This task uses audio recordings of children's story retells, speech-to-text software and language analysis to record, transcribe, analyse and present oral narrative and listening comprehension data back to class teachers. The task has been designed for class teachers' use with the support of SLP or literacy specialists in data interpretation. Teachers are upskilled and supported in order to interpret these data and implement teaching practices for students through online professional learning and development modules, within the context of a broader evidence-based approach to early literacy instruction. This article describes the development of this innovative, culturally relevant, online tool for monitoring children's oral narrative ability and listening comprehension in their first year of school. Three phases of development are outlined, showing the progression of the tool from a researcher-administered task during controlled research trials, to wide-scale implementation with thousands of students throughout New Zealand. The current iteration of the tool uses an automatic speech-recognition system with specifically trained transcription models and support from research assistants to check transcription, then code and analyse the oral narrative. This reduces transcription and analysis time to ~7 min, with a word error rate of around 20%. Future development plans to increase the accuracy of automatic transcription and embed basic language analysis into the tool, with the aim of removing the need for support from research assistants.

Un mot pour un autre ? Analyse et comparaison de huit plateformes de transcription automatique

Cet article compare les fonctionnalités et résultats de huit outils de transcription automatique (Go Transcribe, Happy Scribe, Headliner, Sonix, Video Indexer, Vocalmatic, Vocapia et YouTube), pour des extraits audio de langue française. Une méthodologie innovante, fruit d’un travail interdisciplinaire, est proposée pour comparer les transcriptions. Elle repose sur un assemblage de trois approches complémentaires : (1) une approche quantitative de comparaison de textes à partir d’une métrique couramment employée, le Word Error Rate (WER), (2) une approche fine de classification et compréhension des erreurs générées par les plateformes, et enfin (3) une estimation du potentiel de gain de temps de transcription pour chacun des fichiers et des plateformes. In fine, aucune plateforme ne serait plus efficace que les autres pour l’ensemble des extraits audio mais deux outils se démarquent : Vocapia et Sonix, chacun ayant ses domaines de prédilection. Quel que soit le type de fichier ou de plateforme, un temps de réécoute et de correction reste indispensable à l’issue des traitements, pour un gain de temps final observé pouvant aller jusqu’à 75 % par rapport à une transcription manuelle. Par ailleurs, l’utilisation de ces outils en ligne peut engendrer des problèmes importants liés à la confidentialité et la sécurité des données. Pour finir, nous revenons sur l’expérience de travail interdisciplinaire qui a rendu ce projet possible.

A Music Cognition–Guided Framework for Multi-pitch Estimation

As one of the most important subtasks of automatic music transcription (AMT), multi-pitch estimation (MPE) has been studied extensively for predicting the fundamental frequencies in the frames of audio recordings during the past decade. However, how to use music perception and cognition for MPE has not yet been thoroughly investigated. Motivated by this, this demonstrates how to effectively detect the fundamental frequency and the harmonic structure of polyphonic music using a cognitive framework. Inspired by cognitive neuroscience, an integration of the constant Q transform and a state-of-the-art matrix factorization method called shift-invariant probabilistic latent component analysis (SI-PLCA) are proposed to resolve the polyphonic short-time magnitude log-spectra for multiple pitch estimation and source-specific feature extraction. The cognitions of rhythm, harmonic periodicity and instrument timbre are used to guide the analysis of characterizing contiguous notes and the relationship between fundamental frequency and harmonic frequencies for detecting the pitches from the outcomes of SI-PLCA. In the experiment, we compare the performance of proposed MPE system to a number of existing state-of-the-art approaches (seven weak learning methods and four deep learning methods) on three widely used datasets (i.e. MAPS, BACH10 and TRIOS) in terms of F-measure ({F}_{1}) values. The experimental results show that the proposed MPE method provides the best overall performance against other existing methods.

Does Automatic Speech Recognition (ASR) Have a Role in the Transcription of Indistinct Covert Recordings for Forensic Purposes?

The transcription of covert recordings used as evidence in court is a huge issue for forensic linguistics. Covert recordings are typically made under conditions in which the device needs to be hidden, and so the resulting speech is generally indistinct, with overlapping voices and background noise, and in many cases the acoustic record cannot be analyzed via conventional phonetic techniques (i.e. phonetic segments are unclear, or there are no cues at all present acoustically). In the case of indistinct audio, the resulting transcripts that are produced, often by police working on the case, are often questionable and despite their unreliable nature can be provided as evidence in court. Injustices can, and have, occurred. Given the growing performance of automatic speech recognition (ASR) technologies, and growing reliance on such technologies in everyday life, a common question asked, especially by lawyers and other legal professionals, is whether ASR can solve the problem of what was said in indistinct forensic audio, and this is the main focus of the current paper. The paper also looks at forced alignment, a way of automatically aligning an existing transcriptions to audio. This is an area that needs to be explored in the context of forensic linguistics because transcripts can technically be “aligned” with any audio, making it seem as if it is “correct” even if it is not. The aim of this research is to demonstrate how automatic transcription systems fare using forensic-like audio, and with more than one system. Forensic-like audio is most appropriate for research, because there is greater certainty with what the speech material consists of (unlike in forensic situations where it cannot be verified). Examples of how various ASR systems cope with indistinct audio are shown, highlighting that when a good-quality recording is used ASR systems cope well, with the resulting transcript being usable and, for the most part, accurate. When a poor-quality, forensic-like recording is used, on the other hand, the resulting transcript is effectively unusable, with numerous errors and very few words recognized (and in some cases, no words recognized). The paper also demonstrates some of the problems that arise when forced-alignment is used with indistinct forensic-like audio—the transcript is simply “forced” onto an audio signal giving completely wrong alignment. This research shows that the way things currently stand, computational methods are not suitable for solving the issue of transcription of indistinct forensic audio for a range of reasons. Such systems cannot transcribe what was said in indistinct covert recordings, nor can they determine who uttered the words and phrases in such recordings, nor prove that a transcript is “right” (or wrong). These systems can indeed be used advantageously in research, and for various other purposes, and the reasons they do not work for forensic transcription stems from the nature of the recording conditions, as well as the nature of the forensic context.

A Quality Improvement Project to Improve Experiences of Audio Quality for Remote Attendees of a Ward Round at a London Acute Adult Mixed Psychiatric Inpatient Ward

AimsWe sought to assess the degree, nature and impact of poor audio quality during ward rounds for individuals attending remotely using Microsoft Teams. We aimed to evaluate attendee experiences of audio quality against our expectation that due to the existing ward microphone system using a cardioid polar pickup pattern that attendees would have difficulty hearing all members of the multidisciplinary team, as well as the patient, gathered in the conference room. We also hypothesised that after switching to an omnidirectional sound recording system, we would observe an improvement in attendee satisfaction with audio quality during ward rounds.MethodsIndividuals who had remotely attended ward rounds at Lesney Ward, Oxleas NHS Trust between 01/11/21 and 01/01/22, mainly patient family members and community care co-ordinators, completed a digital feedback questionnaire regarding audio quality. There was no exclusion criteria. Data from Likert scale questions were analysed with descriptive statistical tests (mode and distribution of responses). As minimal demographic data were obtained, inferential statistical tests were not used. Qualitative data were analysed using thematic sorting based on prevalence of themes in the data.ResultsFeedback was provided by three family members, one ward team member and six members of the community mental health team prior to the intervention. Pre-intervention feedback indicated high levels of dissatisfaction with 6/10 respondents reporting they were “dissatisfied” and 1/10 “very dissatisfied”. Only 3/10 of attendees reported being able to hear and understand all individuals physically present in the room. In addition, respondents agreed that the audio quality was poor (modal response “bad”, 6/10), and that the sound quality impacted upon their experience of the ward round (modal response “yes, greatly”, 6/10).The three most common main issues reported by respondents were: people speaking too far from the microphone (7/10), voices sounding muffled (6/10), and too much background noise (4/10). Using their own words, respondents described how the ward round sound quality made them feel. Common themes identified through thematic sorting included: distress, difficulty in understanding information / management plan, ward round prolongation and inability to comprehend the patient or staff.ConclusionIn conclusion, we found that when using an in-built laptop microphone with unidirectional pick-up remote ward round attendee satisfaction was poor, though this improved with the introduction of an omnidirectional system. Key areas for improvement include assessment of optimal positioning for adequate audio pick-up, and the introduction of automatic transcription for individuals with hearing impairments.

What happens to semantics and interaction in a software’s automatic transcriptions? A praxeological gaze on speech-to-text

Cet article appréhende les enjeux relatifs aux transcriptions pour la connaissance scientifique en se focalisant sur des logiciels de transcription automatique appelés speech-to-text. L’article présente le fonctionnement général et les usages des speech-to-text, puis s’attarde sur le travail d’évaluation d’un logiciel spécifique réalisé en tant que sociologue. Cette évaluation a permis d’ouvrir la boîte noire du logiciel en déterminant de façon qualitative les sources de ses erreurs systématiques. Si le logiciel fournit incontestablement un environnement sémantique fiable, il rencontre deux types de problème de nature distincte qui sont confondus dans la littérature computationnelle : ceux générés par les limites des modèles informatiques du logiciel, pouvant se résoudre en améliorant les modèles du système, et ceux liés à la conversation ordinaire, d’ordre praxéologique, qui échappent à la machine. Une comparaison détaillée avec des transcriptions d’analyse de conversation montre comment le logiciel fait disparaître l’interaction de ses transcriptions.

Was geschieht mit Semantik und Interaktion in automatischen Transkriptionen einer Software? Ein praxeologischer Blick auf Speech-to-text

This article addresses the implications of transcriptions for scientific knowledge by focusing on automatic transcription software called speech-to-text. The article presents the general functioning and uses of speech-to-text software, then focuses on the evaluation of a specific software conducted by the author as a social scientist. This evaluation allows one to open the black box of the software by determining in a qualitative way the sources of its systematic errors. While the software undoubtedly provides a reliable semantic environment, it encounters two types of problems of a distinct nature that are intertwined in the computational literature: those generated by the limitations of the software's computational models, which can be solved by improving the system's models, and those related to ordinary conversation of a praxeological nature, which remain beyond the machine's reach. A detailed comparison with transcriptions manually performed in the manner of conversation analysis shows how the software makes interaction disappear from its transcriptions.

Voice Assignment in Vocal Quartets Using Deep Learning Models Based on Pitch Salience

This paper deals with the automatic transcription of four-part, a cappella singing, audio performances. In particular, we exploit an existing, deep-learning based, multiple F0 estimation method and complement it with two neural network architectures for voice assignment (VA) in order to create a music transcription system that converts an input audio mixture into four pitch contours. To train our VA models, we create a novel synthetic dataset by collecting 5381 choral music scores from public-domain music archives, which we make publicly available for further research. We compare the performance of the proposed VA models on different types of input data, as well as to a hidden Markov model-based baseline system. In addition, we assess the generalization capabilities of these models on audio recordings with differing pitch distributions and vocal music styles. Our experiments show that the two proposed models, a CNN and a ConvLSTM, have very similar performance, and both of them outperform the baseline HMM-based system. We also observe a high confusion rate between the alto and tenor voice parts, which commonly have overlapping pitch ranges, while the bass voice has the highest scores in all evaluated scenarios.

Audio Recording Patient-Nurse Verbal Communications in Home Health Care Settings: Pilot Feasibility and Usability Study.

BackgroundPatients’ spontaneous speech can act as a biomarker for identifying pathological entities, such as mental illness. Despite this potential, audio recording patients’ spontaneous speech is not part of clinical workflows, and health care organizations often do not have dedicated policies regarding the audio recording of clinical encounters. No previous studies have investigated the best practical approach for integrating audio recording of patient-clinician encounters into clinical workflows, particularly in the home health care (HHC) setting.ObjectiveThis study aimed to evaluate the functionality and usability of several audio-recording devices for the audio recording of patient-nurse verbal communications in the HHC settings and elicit HHC stakeholder (patients and nurses) perspectives about the facilitators of and barriers to integrating audio recordings into clinical workflows.MethodsThis study was conducted at a large urban HHC agency located in New York, United States. We evaluated the usability and functionality of 7 audio-recording devices in a laboratory (controlled) setting. A total of 3 devices—Saramonic Blink500, Sony ICD-TX6, and Black Vox 365—were further evaluated in a clinical setting (patients’ homes) by HHC nurses who completed the System Usability Scale questionnaire and participated in a short, structured interview to elicit feedback about each device. We also evaluated the accuracy of the automatic transcription of audio-recorded encounters for the 3 devices using the Amazon Web Service Transcribe. Word error rate was used to measure the accuracy of automated speech transcription. To understand the facilitators of and barriers to integrating audio recording of encounters into clinical workflows, we conducted semistructured interviews with 3 HHC nurses and 10 HHC patients. Thematic analysis was used to analyze the transcribed interviews.ResultsSaramonic Blink500 received the best overall evaluation score. The System Usability Scale score and word error rate for Saramonic Blink500 were 65% and 26%, respectively, and nurses found it easier to approach patients using this device than with the other 2 devices. Overall, patients found the process of audio recording to be satisfactory and convenient, with minimal impact on their communication with nurses. Although, in general, nurses also found the process easy to learn and satisfactory, they suggested that the audio recording of HHC encounters can affect their communication patterns. In addition, nurses were not aware of the potential to use audio-recorded encounters to improve health care services. Nurses also indicated that they would need to involve their managers to determine how audio recordings could be integrated into their clinical workflows and for any ongoing use of audio recordings during patient care management.ConclusionsThis study established the feasibility of audio recording HHC patient-nurse encounters. Training HHC nurses about the importance of the audio-recording process and the support of clinical managers are essential factors for successful implementation.

A perceptual measure for evaluating the resynthesis of automatic music transcriptions

This study focuses on the perception of music performances when contextual factors, such as room acoustics and instrument, change. We propose to distinguish the concept of “performance” from the one of “interpretation”, which expresses the “artistic intention”. Towards assessing this distinction, we carried out an experimental evaluation where 91 subjects were invited to listen to various audio recordings created by resynthesizing MIDI data obtained through Automatic Music Transcription (AMT) systems and a sensorized acoustic piano. During the resynthesis, we simulated different contexts and asked listeners to evaluate how much the interpretation changes when the context changes. Results show that: (1) MIDI format alone is not able to completely grasp the artistic intention of a music performance; (2) usual objective evaluation measures based on MIDI data present low correlations with the average subjective evaluation. To bridge this gap, we propose a novel measure which is meaningfully correlated with the outcome of the tests. In addition, we investigate multimodal machine learning by providing a new score-informed AMT method and propose an approximation algorithm for the p-dispersion problem.

A hierarchical reasoning graph neural network for the automatic scoring of answer transcriptions in video job interviews

We address the task of automatically scoring the competency of candidates based on textual features, from the automatic speech recognition transcriptions in the asynchronous video job interviews. The key challenge is to construct the dependency relations and semantic level interaction over each question–answer (QA) pair. However, most recent studies focus on the representation of questions and answers, but ignore the dependency information and interaction between them, which is critical for QA evaluation. In this work, we propose a hierarchical reasoning graph neural network for the automatic assessment of question–answer pairs. Specifically, we construct a sentence-level relational graph neural network to capture the dependency information of sentences in or between the question and the answer. Based on these graphs, we employ a semantic-level reasoning graph attention network to model the interaction states of the current QA session. Finally, we propose a gated recurrent unit encoder to represent the temporal question–answer pairs for the final prediction. Empirical results on CHNAT (a real-world dataset) validate that our proposed model significantly outperforms matching-based benchmark models. Ablation studies and experimental results with 10 random seeds also show the effectiveness and stability of our models.

The possibilities of artificial intelligence in automatic musical transcription of the Tatar folk song

This article is relevant due to the loss of the carriers of folk music that needs to be recorded in digital audio formats and requires music transcription for the subsequent creation of collections for the purposes of scientific research by ethnomusicologists. The study aims at determining the need to use software for the automatic music transcription of audio recordings of folk music. The main research method is the comparative analysis of the music transcription of the Tatar Kryashen songs performed by people and three AI-powered programs (Celemony Melodyne, AudioScore Ultimate and Cubase). Then we compared the scores we prepared and the visual data of three programs: wave, spectral, “piano roll” and traditional music scores. According to five evaluation parameters (the accuracy of displaying a melody, rhythm, key, time signature and subjective assessment), the Cubase program was recognized as the most user-friendly. It is still controversial whether to use artificial intelligence for the music transcription of folk songs since music researchers decide for themselves. The undoubted benefit of the automatic music transcription of folk music is the rapid analysis of audio recordings, the ability to create more music notations in a shorter time, assist in the analysis of fragments that are difficult to hear by ear and restore damaged audio recordings.

Uneven success: automatic speech recognition and ethnicity-related dialects

Addressing racial bias in automatic speech recognition is an area of concern in fields associated with human-computer interaction. Research to date suggests that sociolinguistic variation, namely systematic sources of sociophonetic variation, has yet to be extensively exploited in acoustic model architectures. This paper reports a study that evaluates the performance of one ASR system for a multi-ethnic sample of speakers from the American Pacific Northwest (including Native American, African American, European American and ChicanX speakers). Using a sociophonetic approach to characterizing vocalic and consonantal variation, we ask which dialect features appear to be most challenging for our ASR system. We also ask which error types are particular to the four ethnic dialects sampled. Recordings of both conversational and read speech were coded for a common set of 18 sociophonetic variables with distinct phonetic profiles. Automatic transcription was achieved using CLOx, a custom-built ASR system created for sociolinguistic analysis. Normalized error frequency rates were compared across ethnic samples to evaluate CLOx performance. Nf error rates demonstrate clear differential performance in the ASR system, pointing to racial bias in system output. Specific predictions are made regarding approaches that might be taken to leverage sociophonetic knowledge to improve social dialect-recognition accuracy in ASR systems.

Social Network Data and Epidemiological Intelligence: A Case Study of Avian Influenza

Purpose - Event Based Surveillance (EBS) systems detect and monitor diseases by analysing articles from online news papers and reports from health organizations (e.g. FAO, OIE, etc.). However, they partially integrate data from social networks, even though these data are present in large quantities on the web. The purpose of this study is to exploit social network data, such as Twitter and YouTube, to provide epidemiological and additional information for Avian Influenza surveillance. Methods & Materials - In this context, we propose new text-mining approaches combining lexical rules and statistical approaches in order to normalise textual data from Social Net- work ('h5 n8' - 'H5N8') and to correct errors from YouTube transcriptions (e.g. 'birth u' - 'bird u'). Another challenge consists of extracting epidemiological events automatically by identifying spatial entities (Where?), thematic entities (What?), and temporal information (When?). For this, we extended Named Entity Recognition (NER) tools like spaCy. Results - We collected 100 automatic transcripts of YouTube videos and 268 tweets in English dealing with avian influenza with dedicated API. We obtain encouraging results (i.e. accuracy around 0:6) in order to recognise automatically epidemiological information (e.g. hosts, symptoms, etc.) in textual data con- tents. Extraction of spatial information obtains better results (i.e. accuracy around 0:7). Conclusion – The final objective of this study consists of linking social media data based on these entities with official information from health organisations for the improvement of epidemiological monitoring.

A Pipeline Approach to Context-Aware Handwritten Text Recognition

Despite concerted efforts towards handwritten text recognition, the automatic location and transcription of handwritten text remain a challenging task. Text detection and segmentation methods are often prone to errors, affecting the accuracy of the subsequent recognition procedure. In this paper, a pipeline that locates texts on a page and recognizes the text types, as well as the context of the texts within the detected region, is proposed. Clinical receipts are used as the subject of study. The proposed model is comprised of an object detection neural network that extracts text sequences present on the page regardless of size, orientation, and type (handwritten text, printed text, or non-text). After that, the text sequences are fed to a Residual Network with a Transformer (ResNet-101T) model to perform transcription. Next, the transcribed text sequences are analyzed using a Named Entity Recognition (NER) model to classify the text sequences into their corresponding contexts (e.g., name, address, prescription, and bill amount). In the proposed pipeline, all the processes are implicitly learned from data. Experiments performed on 500 self-collected clinical receipts containing 15,297 text segments reported a character error rate (CER) and word error rate (WER) of 7.77% and 10.77%, respectively.

Metaheuristic adapted convolutional neural network for Telugu speaker diarization

In speech technology, a pivotal role is being played by the Speaker diarization mechanism. In general, speaker diarization is the mechanism of partitioning the input audio stream into homogeneous segments based on the identity of the speakers. The automatic transcription readability can be improved with the speaker diarization as it is good in recognizing the audio stream into the speaker turn and often provides the true speaker identity. In this research work, a novel speaker diarization approach is introduced under three major phases: Feature Extraction, Speech Activity Detection (SAD), and Speaker Segmentation and Clustering process. Initially, from the input audio stream (Telugu language) collected, the Mel Frequency Cepstral coefficient (MFCC) based features are extracted. Subsequently, in Speech Activity Detection (SAD), the music and silence signals are removed. Then, the acquired speech signals are segmented for each individual speaker. Finally, the segmented signals are subjected to the speaker clustering process, where the Optimized Convolutional Neural Network (CNN) is used. To make the clustering more appropriate, the weight and activation function of CNN are fine-tuned by a new Self Adaptive Sea Lion Algorithm (SA-SLnO). Finally, a comparative analysis is made to exhibit the superiority of the proposed speaker diarization work. Accordingly, the accuracy of the proposed method is 0.8073, which is 5.255, 2.45%, and 0.075, superior to the existing works.

PipaSet and TEAS: A Multimodal Dataset and Annotation Platform for Automatic Music Transcription and Expressive Analysis Dedicated to Chinese Traditional Plucked String Instrument Pipa

Music information retrieval (MIR) is developing these years rapidly. As the fundamental MIR tasks, automatic music transcription (AMT) and expressive analysis (EA) are gaining momentum in both Western and non-European music. However, the annotated datasets for non-Eurogenic instruments remain scarce in terms of quantity and feature diversity so that general evaluations and data-driven models on various tasks cannot be well explored. As one of the most popular traditional plucked string instruments in Asia, which is barely studied in the MIR community, pipa has lots of distinctive national and local characteristics including the fake nails, intrinsic pitch shift, rubato, as well as a high diversity of sophisticated playing techniques that greatly enhance the music expressiveness. Our work aims to systematically clarify a creation procedure of a pipa dataset with audio, musical notation and multiview video modalities for traditional Chinese solos. The use of 4-track string vibration signals captured by optical sensors paves a path for high quality annotations. Furthermore, a transcription and Expressiveness Annotation System (TEAS) was transparently implemented to ensure the scalability of dataset. Three expressive analysis approaches in this system were newly proposed and evaluated in this paper. Finally, two AMT models were investigated and a series of the existing and emerging MIR tasks enabled by this dataset were enumerated for the future exploration.

Automatic Lyrics Transcription of Polyphonic Music With Lyrics-Chord Multi-Task Learning

Lyrics are the words that make up a song, while chords are harmonic sets of multiple notes in music. Lyrics and chords are generally essential information in music, i.e. unaccompanied singing vocals mixed with instrumental music, representing important components in polyphonic music. In a traditional lyrics transcription task, we first extract the singing vocals from the polyphonic music and then transcribe the resulting singing vocals, where the two steps are optimized independently. In this paper, we propose novel end-to-end network architectures that are designed to disentangle lyrics from chords in polyphonic music for effective lyrics transcription in a single step, where we consider chords as musical words, analogously to lexical words as lyrics intuitively. We start by studying a single-task lyrics transcriber as the reference baseline and the initial model to develop the multi-task lyrics transcription solutions. The main idea is to take advantage of chord transcription available in the training data through multi-task training to improve lyrics transcription. The experiments show that the proposed multitask lyrics transcriber significantly outperforms other competing solutions, with a word error rate (WER) of 31.82% on a standard test dataset.