Music Information Retrieval Systems Research Articles

Music Information Retrieval using Deep Learning Techniques

Music Information Retrieval (MIR) is gaining attention due to the surge in digital music and the need for efficient search and recommendation systems. Traditional MIR methods rely on hand-crafted features and rule-based systems, limiting their adaptability. Deep Learning (DL) shows promise in automatically extracting complex patterns from raw data. This paper offers an extensive overview of MIR tasks like classification, genre recognition, similarity search, and recommendation, along with DL models like Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), including LSTM and Transformer architectures, tailored for MIR. Challenges such as data scarcity, computational complexity, and interpretability persist, with proposed solutions like data augmentation, transfer learning, and attention mechanisms. Experimental results on benchmark datasets demonstrate DL's superiority in accuracy, scalability, and robustness over traditional methods. Practical examples highlight DL's potential to revolutionize music search, recommendation, and analysis. Emphasizing the importance of large annotated datasets for training high-quality DL models, strategies for data collection, labeling, and preprocessing are outlined. DL offers promising prospects for advancing MIR by addressing its inherent challenges and establishing new performance benchmarks. Further DL development is expected to drive innovation and enhance digital music consumption experiences through MIR systems. Keywords: Deep Learning, Convolution Neural Network (CNNs), Recurrent Neural Network (RNNs), Digital Music Consumption, Genre recognition.

Automated Determination of Music Genre and Linguistic Categorization

The main objective of the project is to build a system that allows users to predict genre as well as language out of some music . The Automated determination of musical genre and linguistic categorization is a machine learning based prediction model which is used to predict the genres and languages of an audio . Musical genres are categorical labels created by humans to characterize piece of music. A musical genre is characterized by common characteristics typically related to the instrumentation, rhythmic structure and harmonic content of the music. The performance and relative importance of the proposed features are investigated by training the statistical pattern recognition classifiers using real world audio collections and using machine learning algorithm.In this project, we built systems called music genre and language classification and the effects of different feature extraction methods and their various combinations were also investigated. This task is of great interest in music information retrieval and recommendation systems. In recent years, several machine learning algorithms have been developed to tackle this task, using features such spectral, temporal, and tonal characteristics of audio signals. Music genre prediction and language classification is a task that involves automatically predicting a piece of music into a specific genre category as well as language based on its audio features. Keywords-machine-learning,music,music genre and language classification

Audio Source Separation as Applied to Vocals-Accompaniment Extraction

This paper presents two systems for extracting the vocals from a musical piece. Vocals extraction finds extensive application in automated lyrics transcription, music information retrieval systems and professional music remixing. However, the current research is dominated by training-intensive models and has not exploited several promising architectures for appreciably accurate yet computationally inexpensive inference. Neural networks can be leveraged to understand the underlying mathematical patterns of human speech and its harmonic overtones as distinguished from those of instruments. A first principles approach behind two distinct model architectures as well as the data processing steps has been described. Semantic segmentation techniques are used to discriminate between the magnitude spectrogram of the vocals and the mixture which is then applied to the first CNN-based model. The second architecture uses gated recurrent units to leverage the unique temporal dependencies of human speech. This model directly performs inference on the entire spectrogram to concurrently learn the frequency distribution of both the vocals as well as the accompaniment. The second architecture is shown to be an improvement upon the first model and intermittently approaches state-of-the-art predictions on the MIR-1K dataset. The novel GRU-based system in particular highlights the feasibility of rapid inference with smaller datasets. These features play a significant role towards source separation research oriented towards the deployment of real-time inference systems.

Music Genre Classification by Machine Learning Algorithms

Working with audio data has been a machine learning issue that has received relatively little attention. With the fast emerging of machine learning and deep learning algorithms, benchmarks for the most recent groundbreaking deep learning research are frequently determined by how well it performs on text and image data. Additionally, models that use text and graphics are where deep learning has made the biggest strides. Speech and audio, equally significant data types, are frequently ignored in this. Audio data has many uses in daily life, ranging from converting spoken words to text, live translation, and employing audio and music for analysis or filtering. A crucial component of music information retrieval systems, content-based music genre classification has featured prominently as a consequence of the emergence of digital music on the Internet. The field of automatically classifying music genres is still incredibly young, and the claimed classification performance levels are currently relatively low.

Vocal92: Audio Dataset With a Cappella Solo Singing and Speech

Singer recognition plays a vital role in music information retrieval systems. Most songs in the singer recognition system are mixed audios of music and voice. In contrast, there is a lack of labeled a cappella solo singing data suitable for singer recognition. Text-independent singer recognition systems successfully encode audio features such as voice pitch, intensity, and timbre to achieve good performance. Most such systems are trained and evaluated using data from music with accompaniment. However, due to the influence of background music, the performance of the singer recognition model was limited. Contrarily, a powerful singer identification system can be trained and evaluated using a cappella solo singing voices with a clear and broad range of qualities. There needs to be labeled clear singing data suitable for singer recognition research. To address this issue, we present Vocal92, a multivariate a cappella solo singing and speech audio dataset spanning around 146.73 hours sourced from volunteers. Furthermore, we use two current state-of-the-art models to construct the singer recognition baseline system. In experiments, the singer recognition model developed by a cappella solo singing data performs well in both single-mode and cross-modal verification data, significantly improving related works. The dataset is accessible to everyone at https://pan.baidu.com/s/1Pn62DHfal2OOZ_5JqgGBdQ with jnz5 as the validation code. For non-commercial use, the dataset will be available free of charge at the IEEE DataPort (after acceptance).

Singer and Genre recognition using deep learning

In this paper, we try to solve the two mostly occurred problems which are genre recognition and singer identification from common pieces of music. For that, we use a system that helps us to achieve our target that is based on a music information retrieval system, in which the system can process our files and the system can generate the desired results. The system is divided into different phases and for the music recognition, first, there is a need to separate the vocal features of the singer from the music clip because, with the background music, it’s slightly difficult to analyze the vocal of the singer, then extract music features, here we work on some of the music features like tempo, RMS, song duration, frequency, dynamic range, and tonality. The RNN (recurrent neural network) has been used for training the model and testing our data set. After the training of the models, the test classifier with different unknown audio files, and the results are 91% accurate for singer identification and 90% for music genre recognition. The system will identify the singer as well as the genre of the song from the same segment of the file. For the data set, the training data used 10 different Indian and Pakistani male and female singers’ audio files in Sindhi, Hindi, and Urdu language for six types of genres like rock, pop, jazz, disco, hip-hop, and classic. The songs were sung by only one singer, the group singing and live concerts music are limitations of our system.

REAL TIME SPEECH EMOTION RECOGNITION USING MACHINE LEARNING

A speech recognition system, which can detect emotions contained in the dataset such as sad, happy, neural, angry, disgust, surprised, fearful and calm expressions. In real time we can use this application in various decisions. Although we are in a pandemic situation all processes are taking place only through online like job interviews, doctor appointments etc. In these cases this application is very useful whether in what state they are and to detect their emotions through speech. Here we are using a library called Librosa. Librosa is a python package for music and audio analysis. It provides the building blocks necessary to concoct music information retrieval systems. It was developed by Brian McFee, assistant professor of music technology and data science at NYU, and creator of Librosa, a python package for music and audio analysis. Librosa upholds a few elements connected with sound records handling and extraction like burden sound from a circle, register of different spectrogram portrayals, symphonious percussive source detachment, conventional spectrogram decay, stacks and translates the sound, Timespace sound handling, successive demonstrating, coordinating consonant percussive partition, beatsimultaneous and some more.

Audio and Music Analysis on the Web using Essentia.js

Open-source software libraries have a significant impact on the development of Audio Signal Processing and Music Information Retrieval (MIR) systems. Despite the abundance of such tools, there is a lack of an extensive and easy-to-use reference library for audio feature extraction on Web clients. In this article, we present Essentia.js, an open-source JavaScript (JS) library for audio and music analysis on both web clients and JS engines. Along with the Web Audio API, it can be used for both offline and real-time audio feature extraction on web browsers. Essentia.js is modular, lightweight, and easy-to-use, deploy, maintain, and integrate into the existing plethora of JS libraries and web technologies. It is powered by a WebAssembly back end cross-compiled from the Essentia C++ library, which facilitates a JS interface to a wide range of low-level and high-level audio features, including signal processing MIR algorithms as well as pre-trained TensorFlow.js machine learning models. It also provides a higher-level JS API and add-on MIR utility modules along with extensive documentation, usage examples, and tutorials. We benchmark the proposed library on two popular web browsers and the Node.js engine, and four devices, including mobile Android and iOS, comparing it to the native performance of Essentia and the Meyda JS library.

Hierarchical multidimensional scaling for the comparison of musical performance styles

Quantification of stylistic differences between musical artists is of academic interest to the music community, and is also useful for other applications such as music information retrieval and recommendation systems. Information about stylistic differences can be obtained by comparing the performances of different artists across common musical pieces. In this article, we develop a statistical methodology for identifying and quantifying systematic stylistic differences among artists that are consistent across audio recordings of a common set of pieces, in terms of several musical features. Our focus is on a comparison of ten different orchestras, based on data from audio recordings of the nine Beethoven symphonies. As generative or fully parametric models of raw audio data can be highly complex, and more complex than necessary for our goal of identifying differences between orchestras, we propose to reduce the data from a set of audio recordings down to pairwise distances between orchestras, based on different musical characteristics of the recordings, such as tempo, dynamics, and timbre. For each of these characteristics, we obtain multiple pairwise distance matrices, one for each movement of each symphony. We develop a hierarchical multidimensional scaling (HMDS) model to identify and quantify systematic differences between orchestras in terms of these three musical characteristics, and interpret the results in the context of known qualitative information about the orchestras. This methodology is able to recover several expected systematic similarities between orchestras, as well as to identify some more novel results. For example, we find that modern recordings exhibit a high degree of similarity to each other, as compared to older recordings.

Cochleogram-based approach for detecting perceived emotions in music

Identifying perceived emotional content of music constitutes an important aspect of easy and efficient search, retrieval, and management of the media. One of the most promising use cases of music organization is an emotion-based playlist, where automatic music emotion recognition plays a significant role in providing emotion related information, which is otherwise, generally unavailable. Based on the importance of the auditory system in emotional recognition and processing, in this study, we propose a new cochleogram-based system for detecting the affective musical content. To effectively simulate the response of the human auditory periphery, the music audio signal is processed by a detailed biophysical cochlear model, thus obtaining an output that closely matches the characteristics of human hearing. In this proposed approach, based on the cochleogram images, which we construct directly from the response of the basilar membrane, a convolutional neural network (CNN) is used to extract the relevant music features. To validate the practical implications of the proposed approach with regard to its possible integration in different digital music libraries, an extensive study was conducted to evaluate the predictive performance of our approach in different aspects of music emotion recognition. The proposed approach was evaluated on publicly available 1000 songs database and the experimental results showed that it performed better in comparison with common musical features (such as tempo, mode, pitch, clarity, and perceptually motivated mel-frequency cepstral coefficients (MFCC)) as well as official ”MediaEval” challenge results on the same reference database. Our findings clearly show that the proposed approach can lead to better music emotion recognition performance and be used as part of a state-of-the-art music information retrieval system.

Analysis and Recognition of Cello Timbre Based on Deep Trust Network Model

Voice color analysis and similarity calculation of music signals are the important research contents of computer music information retrieval system. In this paper, the deep trust network model is applied to the study of musical tone model. The 72-dimensional features of the cello tone are first extracted. Using the wrapper feature selection method, a 14-dimensional optimal feature subset that reflects the tone characteristics is selected, which greatly reduces the complexity of cello tone similarity calculation. In the set, SVR is used to classify and distinguish eight types of tone data, and a recognition accuracy of 62% was achieved, which is verified the feasibility of the tone model.

Robust singer identification of Indian playback singers

Singing voice analysis has been a topic of research to assist several applications in the domain of music information retrieval system. One such major area is singer identification (SID). There has been enormous increase in production of movies and songs in Bollywood industry over the last 50 decades. Surveying this extensive dataset of singers, the paper presents singer identification system for Indian playback singers. Four acoustic features namely—formants, harmonic spectral envelope, vibrato, and timbre—that uniquely describe the singer are extracted from the singing voice segments. Using the combination of these multiple acoustic features, we address the major challenges in SID like the variations in singer’s voice, testing of multilingual songs, and the album effect. Systematic evaluation shows the SID is robust against the variations in singer’s singing style and structure of songs and is effective in identifying the cover songs and singers. The results are investigated on in-house cappella database consisting of 26 singers and 550 songs. By performing dimension reduction of the feature vector and using Support Vector Machine classifier, we achieved an accuracy of 86% using fourfold cross validation process. In addition, performance comparison of the proposed work with other existing approaches reveals the superiority in terms of volume of dataset and song duration.

Contributions of MIR to soundscape ecology. Part I: Potential methodological synergies

We present here the potential synergies between a sub-discipline of music called computational musicology and its associated toolkit Music Information Retrieval (MIR) systems and the current needs of soundscape ecologists who use passive acoustic recorders to collect massive amounts of audio data. These cross-sub-disciplinary synergies fall into three broad methodological categories: (1) automated feature extraction, (2) sound classification and labeling using machine learning, and (3) data visualization. We argue that the use of the MIR toolkit by soundscape ecologists could represent the development of a second generation of soundscape metrics, which are more translatable to policy and natural resource management problems. Two companion articles in this special issue by the authors provide a more detailed examination of the potential for computational musicology's MIR toolkit as applied to questions common to the emerging field of soundscape ecology.

Pattern Based Melody Matching Approach to Music Information Retrieval

Digitization of music and advancements in information technology for sharing information on World Wide Web paved way for its availability in enormous quantities anywhere any time. Rather than retrieving annotated music in response to query given in terms of Metadata such as name of the composer/singer, genre etc modern researchers are challenged towards content based music information retrieval systems (CBMIR). CBMIR systems differ while representing the main melody either as a note sequence or as an analog acoustic signal; the note sequence representation is explored in this research work. Based on the observation that repeating patterns of the note sequences representing the main melody capture the essence of the music object, this research work developed a framework to investigate the feasibility and effectiveness of pattern based melody matching approach to music information retrieval. Experimentation is conducted on a real world dataset of musical objects belonging to South Indian classical music and the performance of the framework is estimated in terms of Mean Reciprocal Ranking.

Combining evidences from magnitude and phase information using VTEO for person recognition using humming

Most of the state-of-the-art speaker recognition system use natural speech signal (i.e., real speech, spontaneous speech or contextual speech) from the subjects. In this paper, recognition of a person is attempted from his or her hum with the help of machines. This kind of application can be useful to design person-dependent Query-by-Humming (QBH) system and hence, plays an important role in music information retrieval (MIR) system. In addition, it can be also useful for other interesting speech technological applications such as human-computer interaction, speech prosody analysis of disordered speech, and speaker forensics. This paper develops new feature extraction technique to exploit perceptually meaningful (due to mel frequency warping to imitate human perception process for hearing) phase spectrum information along with magnitude spectrum information from the hum signal. In particular, the structure of state-of-the-art feature set, namely, Mel Frequency Cepstral Coefficients (MFCCs) is modified to capture the phase spectrum information. In addition, a new energy measure, namely, Variable length Teager Energy Operator (VTEO) is employed to compute subband energies of different time-domain subband signals (i.e., an output of 24 triangular-shaped filters used in the mel filterbank). We refer this proposed feature set as MFCC-VTMP (i.e., mel frequency cepstral coefficients to capture perceptually meaningful magnitude and phase information via VTEO)The polynomial classifier (which is in-principle similar to other discriminatively-trained classifiers such as support vector machine (SVM) with polynomial kernel) is used as the basis for all the experiments. The effectiveness of proposed feature set is evaluated and consistently found to be better than MFCCs feature set for several evaluation factors, such as, comparison with other phase-based features, the order of polynomial classifier, person (speaker) modeling approach (such as, GMM-UBM and i-vector), the dimension of feature vector, robustness under signal degradation conditions, static vs. dynamic features, feature discrimination measures and intersession variability.

The Moodo dataset: Integrating user context with emotional and color perception of music for affective music information retrieval

This paper presents a new multimodal dataset Moodo that can aid the development of affective music information retrieval systems. Moodo’s main novelties are a multimodal approach that links emotional and color perception to music and the inclusion of user context. Analysis of the dataset reveals notable differences in emotion-color associations and their valence-arousal ratings in non-music and music context. We also show differences in ratings of perceived and induced emotions, especially for those with perceived negative connotation, as well as the influence of genre and user context on perception of emotions. By applying an intermediate data fusion model, we demonstrate the importance of user profiles for predictive modeling in affective music information retrieval scenarios.

Task complexity and difficulty in music information retrieval

There has been little research on task complexity and difficulty in music information retrieval (MIR), whereas many studies in the text retrieval domain have found that task complexity and difficulty have significant effects on user effectiveness. This study aimed to bridge the gap by exploring i) the relationship between task complexity and difficulty; ii) factors affecting task difficulty; and iii) the relationship between task difficulty, task complexity, and user search behaviors in MIR. An empirical user experiment was conducted with 51 participants and a novel MIR system. The participants searched for 6 topics across 3 complexity levels. The results revealed that i) perceived task difficulty in music search is influenced by task complexity, user background, system affordances, and task uncertainty and enjoyability; and ii) perceived task difficulty in MIR is significantly correlated with effectiveness metrics such as the number of songs found, number of clicks, and task completion time. The findings have implications for the design of music search tasks (in research) or use cases (in system development) as well as future MIR systems that can detect task difficulty based on user effectiveness metrics.

Turkish Music Genre Classification using Audio and Lyrics Features

Music Information Retrieval (MIR) has become a popular research area in recent years. In this context, researchers have developed music information systems to find solutions for such major problems as automatic playlist creation, hit song detection, and music genre or mood classification. Meta-data information, lyrics, or melodic content of music are used as feature resource in previous works. However, lyrics do not often used in MIR systems and the number of works in this field is not enough especially for Turkish. In this paper, firstly, we have extended our previously created Turkish MIR (TMIR) dataset, which comprises of Turkish lyrics, by including the audio file of each song. Secondly, we have investigated the effect of using audio and textual features together or separately on automatic Music Genre Classification (MGC). We have extracted textual features from lyrics using different feature extraction models such as word2vec and traditional Bag of Words. We have conducted our experiments on Support Vector Machine (SVM) algorithm and analysed the impact of feature selection and different feature groups on MGC. We have considered lyrics based MGC as a text classification task and also investigated the effect of term weighting method. Experimental results show that textual features can also be effective as well as audio features for Turkish MGC, especially when a supervised term weighting method is employed. We have achieved the highest success rate as 99,12\% by using both audio and textual features together.

Query-by-example music information retrieval by score-based genre prediction and similarity measure

A topic of music information retrieval (MIR) field is query-by-example (QBE), which searches a popular music dataset using a user-provided query and aims to find the target song. Since this type of MIR has been generally used in online systems, retrieval time is also as important as accuracy. In this paper, we propose a QBE-based MIR system and investigate the impact of automatic music genre prediction on the performance of it, specifically on perspective of accuracy-time trade-off, using a score-based genre prediction method as well as similarity measures. The proposed system is evaluated on a dataset containing 6000 music pieces from six musical genres, and we show that how much improvement on the performance can be achieved in terms of accuracy and retrieval time, compared with a typical QBE-based MIR system that uses only similarity measures to find the user-desired song.

A query-by-example music retrieval system using feature and decision fusion

An attractive topic of Music Information Retrieval (MIR) is focused on query-by-example (QBE), which receives a user-provided query and aims to find the target song from an associated music dataset. In this paper, we use feature and decision fusion techniques to develop a two-stage accurate and rapid QBE based MIR system. For this purpose, a proposed diverse ensemble of recognizers automatically recognizes the genre of the query in first stage. This diversity is yielded through feature extraction over different frequency bands followed by feature fusion to train the recognizers, and then a decision fusion technique fuses the individual results obtained by members of ensemble. Second stage measures similarity between query and other contents of dataset having the same genre with the query to find the target song. To accomplish this, a distance measure that here is Kullback---Leibler divergence is utilized. In this stage, a genre-adaptive feature extraction method is proposed, and features are also fused by a feature fusion technique. The effectiveness of the feature and decision fusion techniques in our two-stage system (genre recognition; song retrieval) is evaluated experimentally that shows a significant improvement in terms of accuracy and retrieval time in comparison with a system for which those techniques are not applied.