Music Information Retrieval Community Research Articles

Late multimodal fusion for image and audio music transcription

Music transcription, which deals with the conversion of music sources into a structured digital format, is a key problem for Music Information Retrieval (MIR). When addressing this challenge in computational terms, the MIR community follows two lines of research: music documents, which is the case of Optical Music Recognition (OMR), or audio recordings, which is the case of Automatic Music Transcription (AMT). The different nature of the aforementioned input data has conditioned these fields to develop modality-specific frameworks. However, their recent definition in terms of sequence labeling tasks leads to a common output representation, which enables research on a combined paradigm. In this respect, multimodal image and audio music transcription comprises the challenge of effectively combining the information conveyed by image and audio modalities. In this work, we explore this question at a late-fusion level: we study four combination approaches in order to merge, for the first time, the hypotheses regarding end-to-end OMR and AMT systems in a lattice-based search space. The results obtained for a series of performance scenarios–in which the corresponding single-modality models yield different error rates–showed interesting benefits of these approaches. In addition, two of the four strategies considered significantly improve the corresponding unimodal standard recognition frameworks.

PipaSet and TEAS: A Multimodal Dataset and Annotation Platform for Automatic Music Transcription and Expressive Analysis Dedicated to Chinese Traditional Plucked String Instrument Pipa

Music information retrieval (MIR) is developing these years rapidly. As the fundamental MIR tasks, automatic music transcription (AMT) and expressive analysis (EA) are gaining momentum in both Western and non-European music. However, the annotated datasets for non-Eurogenic instruments remain scarce in terms of quantity and feature diversity so that general evaluations and data-driven models on various tasks cannot be well explored. As one of the most popular traditional plucked string instruments in Asia, which is barely studied in the MIR community, pipa has lots of distinctive national and local characteristics including the fake nails, intrinsic pitch shift, rubato, as well as a high diversity of sophisticated playing techniques that greatly enhance the music expressiveness. Our work aims to systematically clarify a creation procedure of a pipa dataset with audio, musical notation and multiview video modalities for traditional Chinese solos. The use of 4-track string vibration signals captured by optical sensors paves a path for high quality annotations. Furthermore, a transcription and Expressiveness Annotation System (TEAS) was transparently implemented to ensure the scalability of dataset. Three expressive analysis approaches in this system were newly proposed and evaluated in this paper. Finally, two AMT models were investigated and a series of the existing and emerging MIR tasks enabled by this dataset were enumerated for the future exploration.

Report on the ISMIR 2020 special session

The 21st edition of the Annual Conference of the International Society for Music Information Retrieval (ISMIR) introduced so-called "special sessions," giving room for discussion on various topics related to music information retrieval (MIR). I report on the activities related to this special session No. 7 with the title "How do we---in MIR research---help artists? Do we?", and summarize the insights gained from the discussion. Being well-aware that the MIR community has come up with contributions that had a positive impact on artists, we particularly discussed the challenges that hinder MIR contributions from being adopted in the field and the problems when MIR innovations with presumably good intentions turn out to have a bad impact. Besides the many challenges, we also came up with ideas how we could move forward and better address artists' needs.

Music Information Retrieval and Contemporary Classical Music: A Successful Failure

This paper is about the story of my relationship, as a contemporary music composer, with computational tools that are situated in the areas of signal processing, machine learning and music information retrieval (MIR). I believe that sharing this story can be useful to the MIR community since it illustrates the problems that can arise when you try to use these techniques in the context of contemporary music creation. Since this is a personal story, I will refer to experiences that I had during about fifteen years of usage of MIR-related technologies. I will show how these technologies tried to (unsuccessfully) shape my musical thinking and why I believe that some of them have come to an end. Finally, I will propose new possible directions for the future of MIR.

Commentary on Shaffer et al.: A cluster analysis of harmony in the McGill Billboard dataset

This short commentary on the target paper by "A cluster analysis of harmony in the McGill Billboard" by Shaffer et al. starts with observing that not all harmonic progressions that are theoretically possible are equally common. Instead, some progressions are more popular than others in popular. In fact, certain harmonic progressions are closely associated with specific styles and sub-genres and it is the aim of the target paper to provide a meaningful classification system for harmonic progression. The commentary identifies several strengths of the target paper, including a nice balance between rigorous empirical work and providing a context and interpretations that are musicologically well-informed. In its critique the commentary points to the limitations of only using harmonic bigrams (i.e. the transitions between two chords) as the empirical data and the missing link to related literature on harmonic modelling in the music information retrieval community.

Efficient Melody Extraction Based on Extreme Learning Machine

Melody extraction is an important task in music information retrieval community and it is unresolved due to the complex nature of real-world recordings. In this paper, the melody extraction problem is addressed in the extreme learning machine (ELM) framework. More specifically, the input musical signal is first pre-processed to mimic the human auditory system. The music features are then constructed by constant-Q transform (CQT), and the concentration strategy is introduced to make use of contextual information. Afterwards, the rough melody pitches are determined by ELM network, according to its pre-trained parameters. Finally, the rough melody pitches are fine-tuned by the spectral peaks around the frame-wise rough pitches. The proposed method can extract melody from polyphonic music efficiently and effectively, where pitch estimation and voicing detection are conducted jointly. Some experiments have been conducted based on three publicly available datasets. The experimental results reveal that the proposed method achieves higher overall accuracies with very fast speed.

Melody Extraction From Polyphonic Music Using Particle Filter and Dynamic Programming

Melody extraction from polyphonic music is one important but challenging task in the music information retrieval community. In this paper, a new melody extraction method based on the particle filter and dynamic programming is proposed. The constant-Q transform is first introduced for multiresolution spectral analysis of polyphonic music. Then, the melody extraction is modeled in the Bayesian filtering framework, and the particle filter is used to get a rough melody contour. Specially, the pitch transition probability of adjacent frames is approximated according to the statistical analysis based on one publicly available dataset, and the likelihood of frame-wise pitches is defined by considering pitch salience, spectral smoothness, and timbre similarity. After that, the preliminary melodic contour obtained by particle filter is smoothed to achieve the frame-wise pitch range limitation. Finally, the dynamic programming is used to accurately track the final melodic contour. The proposed method requires no prior information, and is suitable for both instrumental and vocal melodies. The experimental results show that the performances of the proposed method is robust among four publicly available datasets comparing with the state-of-the-art methods, and it achieves the highest averaged raw pitch accuracy and raw chroma accuracy performances with lower octave errors.

SATIN: a persistent musical database for music information retrieval and a supporting deep learning experiment on song instrumental classification

This paper introduces SATIN, the Set of Audio Tags and Identifiers Normalized. SATIN is a database of 400k audio-related metadata and identifiers that aims at facilitating reproducibility and comparisons among the Music Information Retrieval (MIR) algorithms. The idea is to take advantage of partnerships between scientists and private companies that host millions of tracks. Scientists can send their feature extraction algorithm to companies along SATIN identifiers and retrieve the corresponding features. This procedure allows the MIR community to have access to more tracks for classification purposes. Afterwards, scientists can provide to the MIR community the classification result for each track, which can then be compared with other algorithms results. SATIN thus resolves the major problems of accessing more tracks, managing copyrights locks, saving computation time, and guaranteeing consistency over research databases. We introduce SOFT1, the first Set Of FeaTures extracted by a company thanks to SATIN. We propose a supporting experiment classifying instrumentals and songs to detail a possible use of SATIN. We compare a deep learning approach —that has emerged in recent years in MIR— with a knowledge-based approach.

Effective music searching approach based on tag combination by exploiting prototypical acoustic content

Within the music information retrieval community, many studies and applications have focused on tag-based music categorization. The limitation in employing music tags is the ambiguity of each tag. Thus, a single music tag covers too many sub-categories. To circumvent this, multiple tags can be used simultaneously to specify music clips more precisely. However, in conventional music recommendation systems, this might not be achieved because music clips identified by the system might not be prototypical to both or each tag. In this paper, we propose a new technique for ranking proper tag combinations based on the acoustic similarity of music clips. Based on empirical experiments, proper tag combinations are suggested by our proto-typicality analysis.

Ranking Tag Pairs for Music Recommendation Using Acoustic Similarity

The need for the recognition of music emotion has become apparent in many music information retrieval applications. In addition to the large pool of techniques that have already been developed in machine learning and data mining, various emerging applications have led to a wealth of newly proposed techniques. In the music information retrieval community, many studies and applications have concentrated on tag-based music recommendation. The limitation of music emotion tags is the ambiguity caused by a single music tag covering too many subcategories. To overcome this, multiple tags can be used simultaneously to specify music clips more precisely. In this paper, we propose a novel technique to rank the proper tag combinations based on the acoustic similarity of music clips.

AUDIO DESCRIPTIVE ANALYSIS OF SINGER AND MUSICAL INSTRUMENT IDENTIFICATION IN NORTH INDIAN CLASSICAL MUSIC

Music information retrieval (MIR) has reached to a reasonably stable state after advancement in the Low Level audio Descriptors (LLDs) and feature extraction techniques. The analysis of sound has now become simple by the continuous efforts and research of MIR community in the field of signal processing from last two decades. In north Indian classical music, a singer is accompanied by some instruments such as harmonium, violin or flute. These instruments are tuned in the same musical scale (pitch range) in which the singer is signing. Separate researches have been made in recent past to identify a musical instrument and a singer. In this paper, we have analyzed the low level audio descriptors, for singing voice and musical instrument sound together, that appears to human ear as similar with respect to ‘timbre’, to see if we could treat them same and use identification/ classification routines to classify them into their classes. We have used Hybrid Selection algorithm from wrapper technique(the one that uses classifier also in feature selection process) to identify and extract the features and K-Means and K nearest neighbor classifiers to classify and cross verify the accuracy of classification. The accuracy of classification achieved was 91.1% which clearly proves that musical instruments and singing voice that sounds similar in timbral aspect can be grouped together and classification is possible with mixed database of instruments and singing voices.

Deep Image Features in Music Information Retrieval

Abstract Applications of Convolutional Neural Networks (CNNs) to various problems have been the subject of a number of recent studies ranging from image classification and object detection to scene parsing, segmentation 3D volumetric images and action recognition in videos. CNNs are able to learn input data representation, instead of using fixed engineered features. In this study, the image model trained on CNN were applied to a Music Information Retrieval (MIR), in particular to musical genre recognition. The model was trained on ILSVRC-2012 (more than 1 million natural images) to perform image classification and was reused to perform genre classification using spectrograms images. Harmonic/percussive separation was applied, because it is characteristic for musical genre. At final stage, the evaluation of various strategies of merging Support Vector Machines (SVMs) was performed on well known in MIR community - GTZAN dataset. Even though, the model was trained on natural images, the results achieved in this study were close to the state-of-the-art.

Machine Recognition of Music Emotion

The proliferation of MP3 players and the exploding amount of digital music content call for novel ways of music organization and retrieval to meet the ever-increasing demand for easy and effective information access. As almost every music piece is created to convey emotion, music organization and retrieval by emotion is a reasonable way of accessing music information. A good deal of effort has been made in the music information retrieval community to train a machine to automatically recognize the emotion of a music signal. A central issue of machine recognition of music emotion is the conceptualization of emotion and the associated emotion taxonomy. Different viewpoints on this issue have led to the proposal of different ways of emotion annotation, model training, and result visualization. This article provides a comprehensive review of the methods that have been proposed for music emotion recognition. Moreover, as music emotion recognition is still in its infancy, there are many open issues. We review the solutions that have been proposed to address these issues and conclude with suggestions for further research.

Conductors’ temporal gestures: Spatio temporal cues for visually mediated synchronization.

A growing body of research utilizies computational feature‐extraction methods to quantify both music and music‐related movement. The use of signal processing and visualization techniques, for example, is widespread in the music information retrieval community. At the same time, similar techniques are being applied to the analysis of movement data acquired using motion capture systems and are being increasingly used to quantify both temporal and expressive movements of musicians and others engaged in music‐related activity. One such activity is conducting, and this talk will focus on one specific aspect of this activity: how a conductor conveys the beat to musicians using gesture alone. A number of studies involving a variety of response paradigms, both movement and music feature extraction techniques, and both laboratory and live settings will be presented. Results of these studies suggest that perception of the beat is less related to spatial cues than it is to temporal cues, with periods of high acceleration along the trajectory of a gesture being the best predictor of perceived beat location. In other words, conductors communicate the beat primarily by varying the speed of movement along the trajectory, not by changing the direction of movement.

Music information retrieval seen from the communication technology perspective

Music information retrieval (MIR) is a multidiscipline area. Within this domain one can see various approaches to musical instrument recognition, musical phrase classification, melody classification (e.g., query-by-humming systems), rhythm retrieval, high-level-based music retrieval such as looking for emotions in music or differences in expressiveness, and music search based on listeners’ preferences. One may also find research that tries to correlate low-level descriptor analysis to high-level human perception. Researchers from musical acoustics, musicology, and music domains on one side, and communication technology on the other side, work together within this area. This may foster a framework for broader and deeper comprehension of contributions from all these disciplines and, in addition, translate the automated access to music information, gathered in various forms around the World Wide Web, as a fully understandable process to all participants regardless of their background. The semantic description is becoming a basis of the next Web generation. Several important concepts have been introduced recently by the researchers associated with the MIR community with regard to semantic data processing including techniques for computing with words. In this presentation some aspects related to MIR are briefly reviewed in the context of possible and actual applications of ontology-based approach to this domain.

Music to knowledge: A visual programming environment for the development and evaluation of music information retrieval techniques

The objective of the International Music Information Retrieval Systems Evaluation Laboratory (IMIRSEL) project is the creation of a large, secure corpus of audio and symbolic music data accessible to the music information retrieval (MIR) community for the testing and evaluation of various MIR techniques. As part of the IMIRSEL project, a cross-platform JAVA based visual programming environment called Music to Knowledge (M2K) is being developed for a variety of music information retrieval related tasks. The primary objective of M2K is to supply the MIR community with a toolset that provides the ability to rapidly prototype algorithms, as well as foster the sharing of techniques within the MIR community through the use of a standardized set of tools. Due to the relatively large size of audio data and the computational costs associated with some digital signal processing and machine learning techniques, M2K is also designed to support distributed computing across computing clusters. In addition, facilities to allow the integration of non-JAVA based (e.g., C/C++, MATLAB, etc.) algorithms and programs are provided within M2K. [Work supported by the Andrew W. Mellon Foundation and NSF Grants No. IIS-0340597 and No. IIS-0327371.]