Music Information Retrieval Research Articles

Liszt’s Étude S.136 no.1: audio data analysis of two different piano recordings

In this paper, we review the main signal processing tools of Music Information Retrieval (MIR) from audio data, and we apply them to two recordings (by Leslie Howard and Thomas Rajna) of Franz Liszt’s Étude S.136 no.1, with the aim of uncovering the macro-formal structure and comparing the interpretative styles of the two performers. In particular, after a thorough spectrogram analysis, we perform a segmentation based on the degree of novelty, in the sense of spectral dissimilarity, calculated frame-by-frame via the cosine distance. We then compare the metrical, temporal and timbrical features of the two executions by MIR tools. Via this method, we are able to identify in a data-driven way the different moments of the piece according to their melodic and harmonic content, and to find out that Rajna’s execution is faster and less various, in terms of intensity and timbre, than Howard’s one. This enquiry represents a case study able to show the potentialities of MIR from audio data in supporting traditional music score analyses and in providing objective information for statistically founded musical execution analyses.

A regularization algorithm for local key detection

In the field of music information retrieval, the detection of global key in both popular and classical music has been studied extensively, but local key detection has been studied to a lesser extent, even though modulation is an important component of compositional style. It is particularly challenging to identify key change boundaries correctly. We modeled this task as an optimization problem, that of finding out how to divide a piece into sections in different keys taking into consideration both the quality of the fit between the key and the section and the number of sections. We determined the optimal assignment of key to section using the Krumhansl–Schmuckler algorithm with a slightly modified version of the Krumhansl and Kessler key profile. We included a regularization algorithm in the formulation of our problem to balance the number of sections and avoid superfluous modulations. Using a dataset of 80 randomly chosen pieces of music in a variety of genres and levels of complexity, we compared our algorithm with a hidden Markov model (HMM) to determine which method is better for identifying local key. Our approach yielded significantly more accurate results and suggests future avenues of research.

Perceptual (but not acoustic) features predict singing voice preferences.

Why do we prefer some singers to others? We investigated how much singing voice preferences can be traced back to objective features of the stimuli. To do so, we asked participants to rate short excerpts of singing performances in terms of how much they liked them as well as in terms of 10 perceptual attributes (e.g.: pitch accuracy, tempo, breathiness). We modeled liking ratings based on these perceptual ratings, as well as based on acoustic features and low-level features derived from Music Information Retrieval (MIR). Mean liking ratings for each stimulus were highly correlated between Experiments 1 (online, US-based participants) and 2 (in the lab, German participants), suggesting a role for attributes of the stimuli in grounding average preferences. We show that acoustic and MIR features barely explain any variance in liking ratings; in contrast, perceptual features of the voices achieved around 43% of prediction. Inter-rater agreement in liking and perceptual ratings was low, indicating substantial (and unsurprising) individual differences in participants' preferences and perception of the stimuli. Our results indicate that singing voice preferences are not grounded in acoustic attributes of the voices per se, but in how these features are perceptually interpreted by listeners.

Classification of music genre using support vector machine and convolutional neural network

Music is an art of combining rhythms, melodies, harmony, and form and also an art of expressing content. Almost all people in this world love Music. This depends on the person and what type of Music he likes to listen to. Either it will be Classical, melodious rock Hip-hop, or some other class of genre. The task of classifying music files based on genre is extremely difficult, especially in the area of MIR (Music Information Retrieval). The approach involves deep learning, where CNN and SVM models are trained end-to-end using a spectrogram to classify a signal's genre label. Every person has a different choice. In this paper, a set of Music is classified into different types of genres with the help of machine learning. A set of 1000 audio files containing 10 types of genres, with 100 audio files each has been taken. The audio features have been extracted using the Spectral roll-off, Chroma features, and zero Crossing Rate. Then, the model was predicted using different algorithms like Support Vector Machine (SVM), and Convolutional Neural Networks (CNN). After the model is predicted using the algorithms, CNN outperformed SVM and gave an accuracy of 96.2%.

MCSSME: Multi-Task Contrastive Learning for Semi-supervised Singing Melody Extraction from Polyphonic Music

Singing melody extraction is an important task in the field of music information retrieval (MIR). The development of data-driven models for this task have achieved great successes. However, the existing models have two major limitations: firstly, most of the existing singing melody extraction models have formulated this task as a pixel-level prediction task. The lack of labeling data has limited the model for further improvements. Secondly, the generalization of the existing models are prone to be disturbed by the music genres. To address the issues mentioned above, in this paper, we propose a multi-Task contrastive learning framework for semi-supervised singing melody extraction, termed as MCSSME. Specifically, to deal with data scarcity limitation, we propose a self-consistency regularization (SCR) method to train the model on the unlabeled data. Transformations are applied to the raw signal of polyphonic music, which makes the network to improve its representation capability via recognizing the transformations. We further propose a novel multi-task learning (MTL) approach to jointly learn singing melody extraction and classification of transformed data. To deal with generalization limitation, we also propose a contrastive embedding learning, which strengthens the intra-class compactness and inter-class separability. To improve the generalization on different music genres, we also propose a domain classification method to learn task-dependent features by mapping data from different music genres to shared subspace. MCSSME evaluates on a set of well-known public melody extraction datasets with promising performances. The experimental results demonstrate the effectiveness of the MCSSME framework for singing melody extraction from polyphonic music using very limited labeled data scenarios.

Evolution and Emerging Trends in Musical Information Retrieval: A Comprehensive Review and Future Prospects

The rapid development of the digital music industry has brought challenges for music lovers and researchers. In response to these challenges, the field of Music Information Retrieval (MIR) emerged in the mid-1960s to capture the complicated and multi-layered nature of music. Among the various approaches explored, machine learning methods have shown promise in overcoming the complexity of this interdisciplinary field. This paper is primarily centered on conducting an in-depth review of the current state of research regarding the utilization of machine learning within the field of Music Information Retrieval (MIR). Additionally, it aims to forecast the potential future directions within the MIR industry. Upon extensive literature review, it becomes evident that various machine learning techniques, including Neural Networks (NN), Support Vector Machines (SVM), and K-nearest neighbors (KNN), have found common applicability in this field. Furthermore, this review highlights Convolutional Neural Networks (CNN) and Support Vector Machines (SVM) as potential algorithms poised to shape the future landscape of MIR. The findings of this paper serve to elucidate the direction in which MIR is progressing, offering valuable guidance for forthcoming research and development endeavors. In doing so, it contributes to the continued progress and maturation of the field MIR.

The applications of dynamic time warping in the source separation of percussive sounds

Music source separation (MSS) is the process of splitting various components of a musical piece into individual tracks. This process combines the fields of acoustics and machine learning to extract useful data from music, which assists in a variety of music information retrieval tasks. In the past decade, many methods have been employed to perform MSS with varying levels of success. This research explores the use of dynamic time warping (DTW) for MSS tasks in the time domain. DTW is an algorithm that performs a temporal alignment of two time series to measure their similarity. It is unique in that the algorithm will minimize the Euclidean distance between the two sequences by stretching or compressing them to optimize similarity. This makes DTW a distinctive method for MSS, as it operates entirely in the time domain and classifies sounds without the interference of time warping. The research performed focuses only on the separation of transient, percussive sounds. Measurements taken with a drum kit and a selection of digital drum sounds served as the foundation for tests of the algorithm. The results of this research illustrate the potential of DTW in time domain MSS applications.

TenseMusic: An automatic prediction model for musical tension.

The perception of tension and release dynamics constitutes one of the essential aspects of music listening. However, modeling musical tension to predict perception of listeners has been a challenge to researchers. Seminal work demonstrated that tension is reported consistently by listeners and can be accurately predicted from a discrete set of musical features, combining them into a weighted sum of slopes reflecting their combined dynamics over time. However, previous modeling approaches lack an automatic pipeline for feature extraction that would make them widely accessible to researchers in the field. Here, we present TenseMusic: an open-source automatic predictive tension model that operates with a musical audio as the only input. Using state-of-the-art music information retrieval (MIR) methods, it automatically extracts a set of six features (i.e., loudness, pitch height, tonal tension, roughness, tempo, and onset frequency) to use as predictors for musical tension. The algorithm was optimized using Lasso regression to best predict behavioral tension ratings collected on 38 Western classical musical pieces. Its performance was then tested by assessing the correlation between the predicted tension and unseen continuous behavioral tension ratings yielding large mean correlations between ratings and predictions approximating r = .60 across all pieces. We hope that providing the research community with this well-validated open-source tool for predicting musical tension will motivate further work in music cognition and contribute to elucidate the neural and cognitive correlates of tension dynamics for various musical genres and cultures.

Commentary on: "Perception of Structure in Collective Free Improvisation and its Context Dependency" by Arthur Faraco

This short commentary on Arthur Faraco's target paper investigating the perception of structure in collective free improvisation addresses the question of whether the concept of the "interrupt generator" can be used to explain the segmentations annotated by listeners. In addition, the commentary points to opportunities for further modelling of the musical and perceptual data through the use of modern tools from music information retrieval (MIR).

The Interconnections of Music Structure, Harmony, Melody, Rhythm, and Predictivity

Repetition and structure have a significant place in music theory, but the structure hierarchy and its influences are often ignored in both music analysis and music generation. In this article, we first describe novel algorithms based on repetition to extract music structure hierarchy from a MIDI data set of popular music and show its effectiveness through evaluation. Then, we introduce new data-driven approaches to estimate and validate structural influences in music. Results show that the automatically detected hierarchical repetition structures reveal significant interactions between structure and harmony, melody, rhythm, and predictivity. Different levels of hierarchy interact differently, providing evidence that structural hierarchy plays an important role in our popular music data set beyond simple notions of repetition or similarity. We further study how musical structure has evolved over decades of popular music writing. Finally, we discuss the importance of this work in highlighting roles that structure can play in music analysis, music similarity, music generation, music evaluation, and other music information retrieval tasks.

The Contribution of Music Information Retrieval System Optimization to Music Analysis in the Context of Big Data

The Contribution of Music Information Retrieval System Optimization to Music Analysis in the Context of Big Data

Age- and Sex-Related Differences in Music Requests on a Radio Show for Children

In research on the development of musical preferences, children are often asked either to evaluate musical pieces that have been previously selected by adult experimenters, or to give self-reports on preferences for different musical styles. This study involved analyzing a sample of 1,412 freely and publicly expressed music requests from children aged between 4 and 11 years taken from a German radio program, with regard to age- and sex-specific differences. The music was categorized into genres, as listed on Spotify, and examined using methods of music information retrieval provided by the Spotify Developer application programming interface. Results showed that, at younger ages, the requests were generally more evenly distributed across different genres. Regarding single genres, we observed small positive relationships between age and the likelihood of requesting the genres pop or electro and small negative relationships between age and the likelihood of requesting A capella, German songwriter, indie, theme songs, or children's music. Furthermore, we found that boys requested significantly more rock or hip-hop, whereas girls had a higher tendency to ask for pop. Finally, age- (but no sex-) related differences regarding the Spotify features of valence and liveness of the requested music were found, which were related to the preference for children's music at younger ages. The results thus suggest that previous findings regarding differences between boys and girls and an increasing formation of distinct genre preferences during infancy also apply to single song requests made on a radio show.

Analysis of Audio Differentiation and Sentiment in Korean Songs with Gugak elements : Focusing on Music Information Retrieval and Music Machine Learning

Analysis of Audio Differentiation and Sentiment in Korean Songs with Gugak elements : Focusing on Music Information Retrieval and Music Machine Learning

From Intonation Adjustments to Synchronization of Heart Rate Variability: Singer Interaction in Traditional Georgian Vocal Music

This paper is concerned with how singers of Georgian traditional vocal music interact when singing together. Applying a variety of computational methods from audio signal processing and music information retrieval (MIR), we examine three existing corpora of (field) recordings for manifestations of a high degree of mutual coordination of the singers’ voices. We find numerous examples of harmonically controlled mutual intonation adjustments on both short and long time scales. Furthermore, we believe that the observed differences in melodic and harmonic scales can also be interpreted as (side) effects of the singers’ interaction with the possible goal of achieving harmonic togetherness (or consonance) on the time scale of individual (important) notes. In addition, together with the ensemble Khelkhvavi from Ozurgeti, we conducted an experiment demonstrating the synchronization of singers’ heartbeat rates during the performance of the Gurian song Chven Mshvidoba. The results of our analysis show that a variety of measurable signs of interaction between singers can be observed and documented in existing corpora of Georgian traditional vocal music. Our experience also shows that relevant information about the synchronization of the bodily functions of singers during performances can nowadays be obtained with reasonable technical and logistical effort even in a ‘real world’ framework, thereby allowing us to address questions related to the ‘ecological validity’ of this kind of measurement.

Music emotion classification using a hybrid CNN-LSTM model

The emotional content of music, interwoven with the intricacies of human affect, poses a unique challenge for computational recognition and classification. With the digitalization of music libraries expanding exponentially, there is a pressing need for precise, automated tools capable of navigating and categorizing vast musical repositories based on emotional contexts. This study advances music emotion classification in the field of music information retrieval by developing a deep learning model that accurately predicts emotional categories in music. The goal of this research is to advance the field of music emotion classification by leveraging the capabilities of convolutional neural networks combined with long short-term memory within deep learning frameworks. The contribution of this study is to provide a refined approach to music emotion classification, combining the power of convolutional neural networks and long short-term memory architectures with sophisticated preprocessing of the Emotify dataset for a deeper and more accurate analysis of musical emotions. The research introduces a novel architecture combining Convolutional Neural Networks and Long Short-Term Memory networks designed to capture the intricate emotional nuances in music. The model leverages convolutional neural networks for robust feature detection and Long Short-Term Memory networks for effective sequence learning, addressing the temporal dynamics of musical features. Utilizing the Emotify dataset, comprising tracks annotated with nine emotional features, the study expands the dataset by segmenting each track into 20 parts, thereby enriching the variety of emotional expressions. Techniques like the synthetic minority oversampling technique were implemented to counter dataset imbalance, ensuring equitable representation of various emotions. The spectral characteristics of the samples were analyzed using the Fast Fourier Transform, contributing to a more comprehensive understanding of the data. Through meticulous fine-tuning, including dropout implementation to prevent overfitting and learning rate adjustments, the developed model achieved a notable accuracy of 94.7 %. This high level of precision underscores the model's potential for application in digital music services, recommendation systems, and music therapy. Future enhancements to this music emotion classification system include expanding the dataset and refining the model architecture for even more nuanced emotional analysis

Artificial Intelligence and Music Discovery

ABSTRACT The arrival of ChatGPT has prompted numerous conversations about artificial intelligence’s role in shaping the future of libraries. Our article seeks to contextualize this new technology within music discovery research by defining the historical underpinnings of artificial intelligence in computer music and music information retrieval. We also identify four areas in librarianship research and practice where applications of this technology should be explored; augmenting library interactions with conversational search, creating tools to assist with metadata clean up and creation, integrating music holdings into commercial discovery systems, and improving music discovery platforms. We hope by identifying potential areas of research we inspire others with practical ways they can contribute to the shape of artificial intelligence in music discovery.

Auxiliary self-supervision to metric learning for music similarity-based retrieval and auto-tagging.

In the realm of music information retrieval, similarity-based retrieval and auto-tagging serve as essential components. Similarity-based retrieval involves automatically analyzing a music track and fetching analogous tracks from a database. Auto-tagging, on the other hand, assesses a music track to deduce associated tags, such as genre and mood. Given the limitations and non-scalability of human supervision signals, it becomes crucial for models to learn from alternative sources to enhance their performance. Contrastive learning-based self-supervised learning, which exclusively relies on learning signals derived from music audio data, has demonstrated its efficacy in the context of auto-tagging. In this work, we propose a model that builds on the self-supervised learning approach to address the similarity-based retrieval challenge by introducing our method of metric learning with a self-supervised auxiliary loss. Furthermore, diverging from conventional self-supervised learning methodologies, we discovered the advantages of concurrently training the model with both self-supervision and supervision signals, without freezing pre-trained models. We also found that refraining from employing augmentation during the fine-tuning phase yields better results. Our experimental results confirm that the proposed methodology enhances retrieval and tagging performance metrics in two distinct scenarios: one where human-annotated tags are consistently available for all music tracks, and another where such tags are accessible only for a subset of music tracks.

High-Quality and Reproducible Automatic Drum Transcription from Crowdsourced Data

Within the broad problem known as automatic music transcription, we considered the specific task of automatic drum transcription (ADT). This is a complex task that has recently shown significant advances thanks to deep learning (DL) techniques. Most notably, massive amounts of labeled data obtained from crowds of annotators have made it possible to implement large-scale supervised learning architectures for ADT. In this study, we explored the untapped potential of these new datasets by addressing three key points: First, we reviewed recent trends in DL architectures and focused on two techniques, self-attention mechanisms and tatum-synchronous convolutions. Then, to mitigate the noise and bias that are inherent in crowdsourced data, we extended the training data with additional annotations. Finally, to quantify the potential of the data, we compared many training scenarios by combining up to six different datasets, including zero-shot evaluations. Our findings revealed that crowdsourced datasets outperform previously utilized datasets, and regardless of the DL architecture employed, they are sufficient in size and quality to train accurate models. By fully exploiting this data source, our models produced high-quality drum transcriptions, achieving state-of-the-art results. Thanks to this accuracy, our work can be more successfully used by musicians (e.g., to learn new musical pieces by reading, or to convert their performances to MIDI) and researchers in music information retrieval (e.g., to retrieve information from the notes instead of audio, such as the rhythm or structure of a piece).

Harmonizing minds and machines: survey on transformative power of machine learning in music.

This survey explores the symbiotic relationship between Machine Learning (ML) and music, focusing on the transformative role of Artificial Intelligence (AI) in the musical sphere. Beginning with a historical contextualization of the intertwined trajectories of music and technology, the paper discusses the progressive use of ML in music analysis and creation. Emphasis is placed on present applications and future potential. A detailed examination of music information retrieval, automatic music transcription, music recommendation, and algorithmic composition presents state-of-the-art algorithms and their respective functionalities. The paper underscores recent advancements, including ML-assisted music production and emotion-driven music generation. The survey concludes with a prospective contemplation of future directions of ML within music, highlighting the ongoing growth, novel applications, and anticipation of deeper integration of ML across musical domains. This comprehensive study asserts the profound potential of ML to revolutionize the musical landscape and encourages further exploration and advancement in this emerging interdisciplinary field.

Music-oriented auditory attention detection from electroencephalogram

Music-oriented auditory attention detection (AAD) aims at determining which instrument in polyphonic music a listener is paying attention to by analyzing the listener’s electroencephalogram (EEG). However, the existing linear models cannot effectively mimic the nonlinearity of the human brain, resulting in limited performance. Thus, a nonlinear music-oriented AAD model is proposed in this paper. Firstly, an auditory feature and a musical feature are fused to represent musical sources precisely and comprehensively. Secondly, the EEG is enhanced if music stimuli are presented in stereo. Thirdly, a neural network architecture is constructed to capture nonlinear and dynamic interactions between the EEG and auditory stimuli. Finally, the musical source most similar to the EEG in the common embedding space is identified as the attended one. Experimental results demonstrate that the proposed model outperforms all baseline models. On 1-s decision windows, it reaches accuracies of 92.6% and 81.7% under mono duo and trio stimuli, respectively. Additionally, it can be easily extended to speech-oriented AAD. This work can open up new possibilities for studies on both brain neural activity decoding and music information retrieval.