Music Information Retrieval Research Research Articles

Perceptual and automated estimates of infringement in 40 music copyright cases

Music copyright infringement lawsuits implicate millions of dollars in damages and costs of litigation. There are, however, few objective measures by which to evaluate these claims. Recent music information retrieval research has proposed objective algorithms to automatically detect musical similarity, which might reduce subjectivity in music copyright infringement decisions, but there remains minimal relevant perceptual data despite its crucial role in copyright law. We collected perceptual data from 51 participants for 40 adjudicated copyright cases from 1915–2018 in 7 legal jurisdictions (USA, UK, Australia, New Zealand, Japan, People’s Republic of China, and Taiwan). Each case was represented by three different versions: either full audio, melody only (MIDI), or lyrics only (text). Due to the historical emphasis in legal opinions on melody as the key criterion for deciding infringement, we originally predicted that listening to melody-only versions would result in perceptual judgments that more closely matched actual past legal decisions. However, as in our preliminary study of 17 court decisions (Yuan et al., 2020), our results did not match these predictions. Participants listening to full audio outperformed not only the melody-only condition, but also automated algorithms designed to calculate musical similarity (with maximal accuracy of 83% vs. 75%, respectively). Meanwhile, lyrics-only conditions performed at chance levels. Analysis of outlier cases suggests that music, lyrics, and contextual factors can interact in complex ways difficult to capture using quantitative metrics. We propose directions for further investigation including using larger and more diverse samples of cases, enhanced methods, and adapting our perceptual experiment method to avoid relying on ground truth data only from court decisions (which may be subject to errors and selection bias). Our results contribute data and methods to inform practical debates relevant to music copyright law throughout the world, such as the question of whether, and the extent to which, judges and jurors should be allowed to hear published sound recordings of the disputed works in determining musical similarity. Our results ultimately suggest that while automated algorithms are unlikely to replace human judgments, they may help to supplement them.

Deep Chnet 1-D CNN using raw audio waveform for recognizing traditional Chinese musical instrument

The classification of music information using various deep learning models is increasingly popular in the field of Music Information Retrieval research. However, as most proposed works focus on western music and musical instruments, little attention is given to traditional Chinese music. This paper proposes a 1-D Convolutional Neural Network (1-D CNN) using only raw audio waveform as input, to undertake the task of traditional Chinese musical instruments classification. This paper starts with a review of the current state of research on the related field, then discuss the proposed model and its data in detail, followed by its performance metrics and then a conclusion on the experiment. The result shows that 1-D CNN provides competitive and even superior results when compared to its 2-D versions as well as when compared to traditional models.

Musical Success in the United States and Brazil: Novel Datasets and Temporal Analyses

Music is not only a worldwide essential cultural industry but also one of the most dynamic. The increasing volume of complex music-related data defines new challenges and opportunities for extracting knowledge, benefiting not only different music segments but also the Music Information Retrieval research field. In this article, we assess musical success in the United States and Brazil, two of the biggest music markets in the world. We first introduce MUHSIC and MUHSIC-BR, two novel datasets with enhanced success information that combine chart-related data with acoustic metadata to describe the temporal evolution of musical careers. Then, we use such enriched and curated data to cluster artists according to their success level by considering their high-impact periods (hot streaks). Our results reveal three groups with distinct success behavior over time. Furthermore, Brazil and the US present specific music success patterns regarding artists and genres, reflecting the importance of analyzing regional markets individually.

Improving Audio Chord Estimation by Alignment and Integration of Crowd-Sourced Symbolic Music

Automatic Chord Estimation (ACE) is a fundamental task in Music Information Retrieval (MIR) and has applications in both music performance and MIR research. The task consists of segmenting a music recording or score and assigning a chord label to each segment. Although it has been a task in the annual benchmarking evaluation MIREX for over 10 years, ACE is not yet a solved problem, since performance has stagnated and modern systems have started to tune themselves to subjective training data. We propose DECIBEL, a new ACE system that exploits heterogeneous musical representations, specifically MIDI and tab files, to improve audio-based ACE methods. From an audio file and a set of MIDI and tab files corresponding to the same popular music song, DECIBEL first estimates chord sequences. For audio, state-of-the-art audio ACE methods are used. MIDI files are aligned to the audio, followed by a MIDI chord estimation step. Tab files are transformed into untimed chord sequences and then aligned to the audio. Next, DECIBEL uses data fusion to integrate all estimated chord sequences into one final output sequence. DECIBEL improves all tested state-of-the-art ACE methods by 0.5 to 13.6 percentage points. This result shows that the integration of crowd-sourced annotations from heterogeneous symbolic music representations using data fusion is a suitable strategy for addressing challenging MIR tasks such as ACE.

Semantic‐Based Sports Music Information Fusion and Retrieval in Wireless Sensor Networks

The wireless sensor network has developed rapidly in recent years. It is formed by the intersection of multiple disciplines. It integrates embedded technology, sensor technology, distributed technology, wireless communication technology, and modern networks. It is a brand new information acquisition platform. The characteristics of sensor networks determine that information fusion technology is a hot spot in the research of wireless sensor networks. Information fusion can achieve high performance and low cost in terms of energy and communication, which is of great significance to the research of sensor networks. This paper is aimed at studying the semantic‐based sports music information fusion and retrieval research in wireless sensor networks. WSNs may face various attacks including eavesdropping attacks, replay attacks, Sybil attacks, and DOS attacks. Therefore, they are designing sensor network solutions. It is necessary to consider the network security issues. This article summarizes and analyzes the existing WSN security data fusion solutions for this issue and compares them by classification. This paper proposes methods and theories such as the spatial correlation detection algorithm, CBA algorithm, FTD algorithm, and DFWD algorithm, which enriches the research of information fusion and retrieval in wireless sensor networks, which is of exploratory significance, and it also establishes this problem. The model was studied, and reliable data was obtained. The experimental results of this paper show that when using these methods to diagnose faults in WSN, the correct rate of model diagnosis is higher than 77%.

Report on the ISMIR 2020 special session

The 21st edition of the Annual Conference of the International Society for Music Information Retrieval (ISMIR) introduced so-called "special sessions," giving room for discussion on various topics related to music information retrieval (MIR). I report on the activities related to this special session No. 7 with the title "How do we---in MIR research---help artists? Do we?", and summarize the insights gained from the discussion. Being well-aware that the MIR community has come up with contributions that had a positive impact on artists, we particularly discussed the challenges that hinder MIR contributions from being adopted in the field and the problems when MIR innovations with presumably good intentions turn out to have a bad impact. Besides the many challenges, we also came up with ideas how we could move forward and better address artists' needs.

MTD: A Multimodal Dataset of Musical Themes for MIR Research

Musical themes are essential elements in Western classical music. In this paper, we present the Musical Theme Dataset (MTD), a multimodal dataset inspired by “A Dictionary of Musical Themes” by Barlow and Morgenstern from 1948. For a subset of 2067 themes of the printed book, we created several digital representations of the musical themes. Beyond graphical sheet music, we provide symbolic music encodings, audio snippets of music recordings, alignments between the symbolic and audio representations, as well as detailed metadata on the composer, work, recording, and musical characteristics of the themes. In addition to the data, we also make several parsers and web-based interfaces available to access and explore the different modalities and their relations through visualizations and sonifications. These interfaces also include computational tools, bridging the gap between the original dictionary and music information retrieval (MIR) research. The dataset is of relevance for various subfields and tasks in MIR, such as cross-modal music retrieval, music alignment, optical music recognition, music transcription, and computational musicology.

Dagstuhl ChoirSet: A Multitrack Dataset for MIR Research on Choral Singing

Choral singing is a central part of musical cultures across the world, yet many facets of this widespread form of polyphonic singing are still to be explored. Music information retrieval (MIR) research on choral singing benefits from multitrack recordings of the individual singing voices. However, there exist only few publicly available multitrack datasets on polyphonic singing. In this paper, we present Dagstuhl ChoirSet (DCS), a multitrack dataset of a cappella choral music designed to support MIR research on choral singing. The dataset includes recordings of an amateur vocal ensemble performing two choir pieces in full choir and quartet settings. The audio data was recorded during an MIR seminar at Schloss Dagstuhl using different close-up microphones to capture the individual singers’ voices. In this article, we give detailed insights into all stages of creating DCS: recording process, data preparation, generation of annotations as well as development of suitable interfaces for publicly accessing and reusing the data. Furthermore, we demonstrate the potential of the dataset for MIR research by discussing case studies on choral intonation assessment and multiple fundamental frequency (F0) estimation.

Playing technique classification for bowed string instruments from raw audio

Music instrument playing technique classification based on raw audio is a relatively unexplored area of music information retrieval research. This study systematically investigates the use of traditional audio features augmented by features based on the Hartley transform, used as input to a multiclass support vector machine (SVM) classifier, to identify up to 11 different playing techniques performed on each of the violin, viola, cello, and contrabass. Furthermore, 36- and 44-class joint instrument and playing technique classifiers were developed that achieved macro-average F-measures exceeding 0.88. Our approach expands and improves on the state-of-the-art study, which implemented sparse-coded magnitude and phase-derived spectral features.

Enhancement of Conventional Beat Tracking System Using Teager–Kaiser Energy Operator

Beat detection systems are widely used in the music information retrieval (MIR) research field for the computation of tempo and beat time positions in audio signals. One of the most important parts of these systems is usually onset detection. There is an understandable tendency to employ the most accurate onset detector. However, there are options to increase the global tempo (GT) accuracy and also the detection accuracy of beat positions at the expense of less accurate onset detection. The aim of this study is to introduce an enhancement of a conventional beat detector. The enhancement is based on the Teager–Kaiser energy operator (TKEO), which pre-processes the input audio signal before the spectral flux calculation. The proposed approach is first evaluated in terms of the ability to estimate the GT and beat positions accuracy of given audio tracks compared to the same conventional system without the proposed enhancement. The accuracy of the GT and average beat differences (ABD) estimation is tested on the manually labelled reference database. Finally, this system is used for analysis of a string quartet music database. Results suggest that the presence of the TKEO lowers onset detection accuracy but also increases the GT and ABD estimation. The average deviation from the reference GT in the reference database is 9.99 BPM (11.28%), which improves the conventional methodology, where the average deviation is 18.19 BPM (17.74%). This study has a pilot character and provides some suggestions for improving the beat tracking system for music analysis.

Data analysis on music classification system and creating a sentiment word dictionary for Kokborok language

This work shows the development of a lexicon for a poorly resourced language, namely Kokborok. Kokborok is a regional language of North East India and offers an entirely new base for research in music information retrieval (MIR) field. We first create a sentimental word dictionary known as lexicons to develop a polarity classification system. It is a text analysis work involving two types of lyrical features that are: ‘text stylistic feature’, and the features were taken out from the newly developed dictionary. We have also shown the comparative analysis with a various subset of music database based on their accuracy rate. After the system development, the experimental/simulations were done, and the results have been computationally analyzed. We performed linear extrapolation of the data taken by both the feature set, thus developing a dictionary. Text stylistic (TS) features have been observed to converge, at 52 and 39 percent respectively for the number of songs tending to infinity. It has been found that at present, it might be better to increase the features from the dictionary since it gives better accuracy for low resource language Kokborok.

Open-Source Practices for Music Signal Processing Research: Recommendations for Transparent, Sustainable, and Reproducible Audio Research

In the early years of music information retrieval (MIR), research problems were often centered around conceptually simple tasks, and methods were evaluated on small, idealized data sets. A canonical example of this is genre recognition-i.e., Which one of n genres describes this song?-which was often evaluated on the GTZAN data set (1,000 musical excerpts balanced across ten genres) [1]. As task definitions were simple, so too were signal analysis pipelines, which often derived from methods for speech processing and recognition and typically consisted of simple methods for feature extraction, statistical modeling, and evaluation. When describing a research system, the expected level of detail was superficial: it was sufficient to state, e.g., the number of mel-frequency cepstral coefficients used, the statistical model (e.g., a Gaussian mixture model), the choice of data set, and the evaluation criteria, without stating the underlying software dependencies or implementation details. Because of an increased abundance of methods, the proliferation of software toolkits, the explosion of machine learning, and a focus shift toward more realistic problem settings, modern research systems are substantially more complex than their predecessors. Modern MIR researchers must pay careful attention to detail when processing metadata, implementing evaluation criteria, and disseminating results.

“… there is no reason why it should ever stop”: Large-scale algorithmic composition

This article surveys some new directions in algorithmic composition, with a special focus on the mass scale of content generation now within reach. Interest in algorithmic composition has recently flourished, including a fashion for deep learning, and commercial offshoots in apps and browser software. Whilst there is much to be excited about in recent developments, we critically survey some current directions, and likely future initiatives, of the field. In particular, the influence of and potential within music information retrieval research is highlighted, corresponding to attempts to bridge between audio machine listening and large corpus training for algorithmic composition. The scenario of mass generation is further considered, including a practical experiment in creating a billion melody data set with accompanying source code.

Corposing a History of Electronic Music

A current research project led by the author has collated nearly 2,000 historic electronic music works for the purposes of musicology; nonetheless, this collection is highly amenable to composition. New pieces can be realized by rendering a selected chronology of electronic music history. The context is a wider field of compositional endeavor in “corposition” over large audio databases especially opened up by new research in music information retrieval.

Toward Studying Music Cognition with Information Retrieval Techniques: Lessons Learned from the OpenMIIR Initiative.

As an emerging sub-field of music information retrieval (MIR), music imagery information retrieval (MIIR) aims to retrieve information from brain activity recorded during music cognition–such as listening to or imagining music pieces. This is a highly inter-disciplinary endeavor that requires expertise in MIR as well as cognitive neuroscience and psychology. The OpenMIIR initiative strives to foster collaborations between these fields to advance the state of the art in MIIR. As a first step, electroencephalography (EEG) recordings of music perception and imagination have been made publicly available, enabling MIR researchers to easily test and adapt their existing approaches for music analysis like fingerprinting, beat tracking or tempo estimation on this new kind of data. This paper reports on first results of MIIR experiments using these OpenMIIR datasets and points out how these findings could drive new research in cognitive neuroscience.

The mood of Chinese Pop music: Representation and recognition

Music mood recognition (MMR) has attracted much attention in music information retrieval research, yet there are few MMR studies that focus on non‐Western music. In addition, little has been done on connecting the 2 most adopted music mood representation models: categorical and dimensional. To bridge these gaps, we constructed a new data set consisting of 818 Chinese Pop (C‐Pop) songs, 3 complete sets of mood annotations in both representations, as well as audio features corresponding to 5 distinct categories of musical characteristics. The mood space of C‐Pop songs was analyzed and compared to that of Western Pop songs. We also explored the relationship between categorical and dimensional annotations and the results revealed that one set of annotations could be reliably predicted by the other. Classification and regression experiments were conducted on the data set, providing benchmarks for future research on MMR of non‐Western music. Based on these analyses, we reflect and discuss the implications of the findings to MMR research.

The Digital Music Lab

In musicology and music research generally, the increasing availability of digital music, storage capacities, and computing power enable and require new and intelligent systems. In the transition from traditional to digital musicology, many techniques and tools have been developed for the analysis of individual pieces of music, but large-scale music data that are increasingly becoming available require research methods and systems that work on the collection-level and at scale. Although many relevant algorithms have been developed during the past 15 years of research in Music Information Retrieval, an integrated system that supports large-scale digital musicology research has so far been lacking. In the Digital Music Lab (DML) project, a collaboration among music librarians, musicologists, computer scientists, and human-computer interface specialists, the DML software system has been developed that fills this gap by providing intelligent large-scale music analysis with a user-friendly interactive interface supporting musicologists in their exploration and enquiry. The DML system empowers musicologists by addressing several challenges: distributed processing of audio and other music data, management of the data analysis process and results, remote analysis of data under copyright, logical inference on the extracted information and metadata, and visual web-based interfaces for exploring and querying the music collections. The DML system is scalable and based on Semantic Web technology and integrates into Linked Data with the vision of a distributed system that enables music research across archives, libraries, and other providers of music data. A first DML system prototype has been set up in collaboration with the British Library and I Like Music Ltd. This system has been used to analyse a diverse corpus of currently 250,000 music tracks. In this article, we describe the DML system requirements, design, architecture, components, and available data sources, explaining their interaction. We report use cases and applications with initial evaluations of the proposed system.

Vowel duration dependent hidden Markov model for automatic lyrics recognition

Recently, due to the spread of music distribution service, a large amount of music is available on the Internet. Accordingly, it is generally increasing the demand of music information retrieval (MIR). In the field of MIR research, there are several researches to extract meaningful information from music audio signals. However, automatic lyrics recognition is still a challenging problem because the variation of singing voice is much larger than that of speaking voice and a large database of singing voice is not available. In the relevant study, lyrics recognition was performed by extending the framework of speech recognition using hidden Markov model (HMM). However, accuracy rate was not sufficient. To recognize singing voice precisely, one promising approach is utilizing musical features. This study considers the task of recognizing syllable from a cappella singing voice. To respond to the variation of the length of a phoneme, we construct the duration dependent HMM. A large database of singing voice is essential for training the acoustic model. We use synthetic singing voice by HMM based singing voice synthesis system to solve the lack of the database of a cappella singing voice. We confirmed the effectiveness of our method.

Combining visual and acoustic features for music genre classification

Since musical genre is one of the most common ways used by people for managing digital music databases, music genre recognition is a crucial task, deep studied by the Music Information Retrieval (MIR) research community since 2002. In this work we present a novel and effective approach for automated musical genre recognition based on the fusion of different set of features. Both acoustic and visual features are considered, evaluated, compared and fused in a final ensemble which show classification accuracy comparable or even better than other state-of-the-art approaches. The visual features are locally extracted from sub-windows of the spectrogram taken by Mel scale zoning: the input signal is represented by its spectrogram which is divided in sub-windows in order to extract local features; feature extraction is performed by calculating texture descriptors and bag of features projections from each sub-window; the final decision is taken using an ensemble of SVM classifiers. In this work we show for the first time that a bag of feature approach can be effective in this problem. As the acoustic features are concerned, we propose an ensemble of heterogeneous classifiers for maximizing the performance that could be obtained starting from the acoustic features. First timbre features are obtained from the audio signal, second some statistical measures are calculated from the texture window and the modulation spectrum, third a feature selection is executed to increase the recognition performance and decrease the computational complexity. Finally, the resulting descriptors are classified by fusing the scores of heterogeneous classifiers (SVM and Random subspace of AdaBoost). The experimental evaluation is performed on three well-known databases: the Latin Music Database (LMD), the ISMIR 2004 database and the GTZAN genre collection. The reported performance of the proposed approach is very encouraging, since they outperform other state-of-the-art approaches, without any ad hoc parameter optimization (i.e. using the same ensemble of classifiers and parameters setting in all the three datasets). The advantage of using both visual and audio features is also proved by means of Q-statistics, which confirms that the two sets of features are partially independent and they are suitable to be fused together in a heterogeneous system. The MATLAB code of the ensemble of classifiers and for the visual features extraction will be publicly available (see footnote 1) to other researchers for future comparisons. The code for acoustic features is not available since it is used in a commercial system.

Toward a universal, meta-theoretical framework for music information classification and retrieval

Purpose– The purpose of this paper is to present a new framework for representing music for information retrieval that emphasizes socio-cultural aspects of music.Design/methodology/approach– Philosophical and theoretical concepts related to the nature of music, aboutness, musical works are explored as they inform how music is represented. Multidisciplinary perspectives on music information representation, classification, and retrieval provide insight into how information science can better accommodate music information within its disciplinary boundaries.Findings– A new term, music information object (MIO), is presented and defined. Downie’s (2003) theoretical statements are reconceptualized into a theory of representational incompleteness and three meta-classes for music information object representation.Practical implications– This new framework incorporates more dimensions of music representation than existing frameworks allow and can facilitate comparisons between classifications of MIO representations by music practitioners, scholars, and system developers.Originality/value– The meta-classes form a much-needed theoretical framework for classifying and defining MIOs from any musical tradition for retrieval. This fills a gap in music information retrieval research, which lacks a theoretical framework that can accommodate musics from all traditions without attempting to organize them according to a western-centered understanding.