Analysis Of Musical Performance Research Articles

Audio Matters Too! Enhancing Markerless Motion Capture with Audio Signals for String Performance Capture

In this paper, we touch on the problem of markerless multi-modal human motion capture especially for string performance capture which involves inherently subtle hand-string contacts and intricate movements. To fulfill this goal, we first collect a dataset, named String Performance Dataset (SPD), featuring cello and violin performances. The dataset includes videos captured from up to 23 different views, audio signals, and detailed 3D motion annotations of the body, hands, instrument, and bow. Moreover, to acquire the detailed motion annotations, we propose an audio-guided multi-modal motion capture framework that explicitly incorporates hand-string contacts detected from the audio signals for solving detailed hand poses. This framework serves as a baseline for string performance capture in a completely markerless manner without imposing any external devices on performers, eliminating the potential of introducing distortion in such delicate movements. We argue that the movements of performers, particularly the sound-producing gestures, contain subtle information often elusive to visual methods but can be inferred and retrieved from audio cues. Consequently, we refine the vision-based motion capture results through our innovative audio-guided approach, simultaneously clarifying the contact relationship between the performer and the instrument, as deduced from the audio. We validate the proposed framework and conduct ablation studies to demonstrate its efficacy. Our results outperform current state-of-the-art vision-based algorithms, underscoring the feasibility of augmenting visual motion capture with audio modality. To the best of our knowledge, SPD is the first dataset for musical instrument performance, covering fine-grained hand motion details in a multi-modal, large-scale collection. It holds significant implications and guidance for string instrument pedagogy, animation, and virtual concerts, as well as for both musical performance analysis and generation. Our code and SPD dataset are available at https://github.com/Yitongishere/string_performance.

Virtual Instrument Performances (VIP): A Comprehensive Review

AbstractDriven by recent advancements in Extended Reality (XR), the hype around the Metaverse, and real‐time computer graphics, the transformation of the performing arts, particularly in digitizing and visualizing musical experiences, is an ever‐evolving landscape. This transformation offers significant potential in promoting inclusivity, fostering creativity, and enabling live performances in diverse settings. However, despite its immense potential, the field of Virtual Instrument Performances (VIP) has remained relatively unexplored due to numerous challenges. These challenges arise from the complex and multi‐modal nature of musical instrument performances, the need for high precision motion capture under occlusions including the intricate interactions between a musician's body and fingers with instruments, the precise synchronization and seamless integration of various sensory modalities, accommodating variations in musicians' playing styles, facial expressions, and addressing instrument‐specific nuances. This comprehensive survey delves into the intersection of technology, innovation, and artistic expression in the domain of virtual instrument performances. It explores musical performance multi‐modal databases and investigates a wide range of data acquisition methods, encompassing diverse motion capture techniques, facial expression recording, and various approaches for capturing audio and MIDI data (Musical Instrument Digital Interface). The survey also explores Music Information Retrieval (MIR) tasks, with a particular emphasis on the Musical Performance Analysis (MPA) field, and offers an overview of various works in the realm of Musical Instrument Performance Synthesis (MIPS), encompassing recent advancements in generative models. The ultimate aim of this survey is to unveil the technological limitations, initiate a dialogue about the current challenges, and propose promising avenues for future research at the intersection of technology and the arts.

Cognitive neuroscience and music education: Relationships and interactions

This article examines the relationship between cognitive neuroscience and music education, evaluating the impact of cognitive neuroscience on musical learning, performance, and teaching strategies. The first section establishes the purpose and scope of the article, followed by an emphasis on the foundations of cognitive neuroscience in the second section. Cognitive neuroscience is explored in terms of music perception, mental processes, and fundamental concepts related to learning and memory. The third section, focusing on cognitive processes in music education, delves into the cognitive foundations of musical knowledge, addressing topics such as melody and rhythm perception and cognitive development in instrumental education. The fourth section discusses the neuroscientific analysis of musical performance, stage fright, brain activity, creativity, and mental processes. The fifth section discusses cognitive neuroscience-based teaching models, the neuroscientific foundations of student-centred approaches, and the role of technology in music education. The sixth section highlights future perspectives and research directions, recommendations for future studies, contributions of new cognitive neuroscience findings to music education, current developments in the intersection of music and cognitive neuroscience, and suggestions for future research. In the concluding section, the summarised article's significant findings and contributions are emphasised, drawing attention to potential areas for future examination and application. This article aims to serve as a comprehensive resource for readers interested in understanding the connections between cognitive neuroscience and music education, providing a foundation for further research in this field.

The “mirroring” method: a case study to enhance the nexus between music performance practice and artistic research in higher music education at European conservatories

After the Bologna Declaration (1999) and the institution of artistic research in music (i.e., PhD in the Arts) in European conservatories, higher music education is facing a revolution. The renewed study program within today’s international music scene is oriented to give students the possibility to be trained and prepared as both professional musicians and/or artist-researchers. Therefore, the current debate at European conservatories is strongly focused on how to activate a nexus between performance practice and artistic research within bachelor’s and master's programmes. This paper presents and discusses a case study of a workshop in music performance practice integrated in the Research Practice Program at Royal Conservatory of Antwerp. It consists of a series of teaching modules inspired by the mirroring method I developed during my PhD at University Ghent – Institute for Psychoacoustic and Electronic Music (IPEM) - and I implemented, as a postdoctoral artist-researcher, in my teaching at the Royal Conservatory of Antwerp (2021-2023). The goal of this method is to assist students in growing up their own artistic identity and trajectory and become professional performers with a possible path on artistic research in music. Students are guided on how to structure music performance practice as a research activity through a systematic documentation, analysis, and dissemination of their own creative process. Outputs can be identified in the improvement of students’ body awareness while playing in relation to score analysis and the achievements of research questions that arise from their reflection in and on their practice. An impact of this approach is retrieved in the elaboration of bachelor’s and master’s theses that can ultimately constitute a bridge for future artistic research proposals. Keywords. Music Performance Practice, Artistic Research in Music, Higher Music Education, Music Performance Analysis, Music Analysis.

Classification of Interpretation Differences in String Quartets Based on the Origin of Performers

Music Information Retrieval aims at extracting relevant features from music material, while Music Performance Analysis uses these features to perform semi-automated music analysis. Examples of interdisciplinary cooperation are, for example, various classification tasks—from recognizing specific performances, musical structures, and composers to identifying music genres. However, some classification problems have not been addressed yet. In this paper, we focus on classifying string quartet music interpretations based on the origin of performers. Our dataset consists of string quartets from composers A. Dvořák, L. Janáček, and B. Smetana. After transferring timing information from reference recordings to all target recordings, we apply feature selection methods to rank the significance of features. As the main contribution, we show that there are indeed origin-based tempo differences, distinguishable by measure durations, by which performances may be identified. Furthermore, we train a machine learning classifier to predict the performers’ origin. We evaluate three different experimental scenarios and achieve higher classification accuracy compared to the baseline using synchronized measure positions.

Analysis of Musical Performance and Characterization in Chinese Opera

Analysis of Musical Performance and Characterization in Chinese Opera

PQG-A2SA: Performance Quantification Guided Audio-to-Score Alignment for Orchestral Music

Audio-to-score alignment is a multi-modal task that aims at generating an accurate mapping between symbolic and signal-level representations of musical signals, which is important for music performance analysis and retrieval. Among numerous music genres, orchestral music is a category of music with complex performance characteristics such as multi-instrument, non-percussive instrument and music expressiveness. However, previous methods do not take sufficient account of the performance characteristics of orchestral music, leading to limitations in alignment accuracy on orchestral music of these methods. To solve this problem, we present a performance quantification guided audio-to-score alignment (PQG-A2SA) method with high alignment accuracy for orchestral music at note-level. Specially, the PQG-A2SA contains two parts, namely an Inter Onset Interval (IOI) guided conditionally-constrained Dynamic Time Wrapping (DTW) and an articulation guided onset and offset detection. Different from the previous work, the IOI-guided conditionally-constrained DTW is designed to achieve a preliminary mapping between symbolic and chord-level representations of musical signals. In the second module, the onset and offset detection model under different musical articulations are established, thus refining the alignment results. We provide extensive experimental validation and analysis of our method. Our PQG-A2SA method can improve 9.0% in onset align rate and 17.5% in offset align rate at most compared with the state-of-the-art methods.

Sonic Visualiser: Visualisation, Analysis, and Annotation of Music Audio Recordings

Sonic Visualiser: Visualisation, Analysis, and Annotation of Music Audio Recordings

An Interdisciplinary Review of Music Performance Analysis

A musical performance renders an acoustic realization of a musical score or other representation of a composition. Different performances of the same composition may vary in terms of performance parameters such as timing or dynamics, and these variations may have a major impact on how a listener perceives the music. The analysis of music performance has traditionally been a peripheral topic for the MIR research community, where often a single audio recording is used as representative of a musical work. This paper surveys the field of Music Performance Analysis (MPA) from several perspectives including the measurement of performance parameters, the relation of those parameters to the actions and intentions of a performer or perceptual effects on a listener, and finally the assessment of musical performance. This paper also discusses MPA as it relates to MIR, pointing out opportunities for collaboration and future research in both areas.

Koncertowe sacrum i profanum – teatralizacja przestrzeni scenicznej na przykładzie zespołu Rammstein

W artykule zwrócono uwagę na problem teatralizacji przestrzeni podczas występów scenicznych grupy Rammstein. Twórczość sceniczna zespołu wykorzystuje środki wyrazu artystycznego, jakie można znaleźć podczas przedstawień teatralnych. Począwszy od oświetlenia, poprzez kostiumy, dekoracje, aż do detali, grupa wykazuje dbałość o szczegóły swoich występów. Niejednokrotnie muzycy sięgają do klipów wideo, przenosząc ich elementy do przestrzeni scenicznej. Niecodzienny, świąteczny charakter wydarzenia, jakim jest koncert zespołu Rammstein, stwarza swoiste przestrzenie, które można określić jako sacrum. W artykule dokonano analizy spektakli muzycznych w świetle teorii Johana Huizingi, Beaty Hoffmann, Wojciecha Siwaka oraz Theodora Adorna i Maxa Horkheimera.

Creating a Multitrack Classical Music Performance Dataset for Multimodal Music Analysis: Challenges, Insights, and Applications

We introduce a dataset for facilitating audio-visual analysis of music performances. The dataset comprises 44 simple multi-instrument classical music pieces assembled from coordinated but separately recorded performances of individual tracks. For each piece, we provide the musical score in MIDI format, the audio recordings of the individual tracks, the audio and video recording of the assembled mixture, and ground-truth annotation files including frame-level and note-level transcriptions. We describe our methodology for the creation of the dataset, particularly highlighting our approaches for addressing the challenges involved in maintaining synchronization and expressiveness. We demonstrate the high quality of synchronization achieved with our proposed approach by comparing the dataset with existing widely-used music audio datasets. We anticipate that the dataset will be useful for the development and evaluation of existing music information retrieval (MIR) tasks, as well as for novel multi-modal tasks. We benchmark two existing MIR tasks (multi-pitch analysis and score-informed source separation) on the dataset and compare with other existing music audio datasets. Additionally, we consider two novel multi-modal MIR tasks (visually informed multi-pitch analysis and polyphonic vibrato analysis) enabled by the dataset and provide evaluation measures and baseline systems for future comparisons (from our recent work). Finally, we propose several emerging research directions that the dataset enables.

Audiovisual Analysis of Music Performances: Overview of an Emerging Field

In the physical sciences and engineering domains, music has traditionally been considered an acoustic phenomenon. From a perceptual viewpoint, music is naturally associated with hearing, i.e., the audio modality. Moreover, for a long time, the majority of music recordings were distributed through audio-only media, such as vinyl records, cassettes, compact discs, and mp3 files. As a consequence, existing automated music analysis approaches predominantly focus on audio signals that represent information from the acoustic rendering of music.

Musical performance analysis in terms of emotions it evokes

Finding pieces with a similar emotional distribution throughout the same composition was the aim of this work. A comparative analysis of musical performances by using emotion tracking was proposed. A dimensional approach of dynamic music emotion recognition was used in the analysis. Music data annotation and regressor training were done. Values of arousal and valence, predicted by regressors, were used to compare performances. The obtained results confirm the validity of the assumption that tracking and analyzing the values of arousal and valence over time in different performances of the same composition can be used to indicate their similarities. Detailed results of analyzing different performances of Prelude No.1 by Frédéric Chopin were presented. They enabled to find the most similar performances to the performance by Arthur Rubinstein, for example. The author found which performances of the same composition were closer to each other and which were quite distant in terms of the shaping of arousal and valence over time. The presented method gives access to knowledge on the shaping of emotions by a performer, which had previously been available only to music professionals.

Representation of musical performance “grammar” using probabilistic graphical models

As a versatile data modeling tool, probabilistic graphical model can be applied to model the complex dependency structures encoded in the contextual “grammar” of music performance. The musical performance grammar here refers to the relational structures of the sonic features extracted from music performances. In the existing literature, the data structure of musical expressive grammar is usually modeled as rule list, following the grammatical format of natural language processing applications. In this work, we apply the representation format of probabilistic graphical model to musical performance features to extend the conventional rule-list format. We choose probabilistic graphical model as an “upgraded” representation for two reasons. First, probabilistic graphical model provides enhanced representation capability of relational structures. This feature enables us to model the complex dependency structure that the conventionally rule list cannot handle. Second, the graphical format of probabilistic graphical model provides an intuitive human-data interface and allows in-depth data visualization, analysis, and interaction. We include the representation and analysis examples of musical performance grammar obtained from both manual analysis and automatic induction. We also implemented interpretation tools that interface the rule-list format and the probabilistic graphical model format to enable detailed comparison with existing results of musical performance analysis.

Inferring musical structure through bodily gestures

Despite the idiosyncrasies present in bodily motion during musical performances, it is suggested that these movements are not incidental but instead contribute to the meaning of performed music. Nine pianists performing two Chopin Preludes are recorded using Vicon motion capture, in synchrony with audio and MIDI recordings. Using principal components analysis (PCA) on the recorded upper body motion, motion profiles for each pianist are formed from the weighted combinations of these components. This measure of global body movement produces a comparable measure across performers and pieces, whilst keeping the performers’ interpretative gestures in view of their whole concept of the piece intact. This study proposes that performers use these motions as a way of corporeally manifesting their interpretative choices. It is suggested that their overall body motion is underscored by such structures as interpreted from the written music. Results show evident underlying musical structure across performers’ motion profiles. These profiles are seen to be constructed from functions of harmonic and melodic relationships in the score. These findings hold significant implications for research into music perception, pedagogy and development of computational technologies for the analysis of music performance.

A Phylogenetic Approach to Music Performance Analysis

This paper presents a novel algorithmic approach to music performance analysis. Previous attempts to use algorithmic tools in this field focused typically on tempo and dynamics alone. We base our analysis on ten different performance categories (such as bowing, vibrato and durations). We adapt phylogenetic analysis tools to resolve the inherent inconsistencies between these categories, and describe the relationships between performances. Taking samples from 29 different performances of two pieces from Bach's sonatas for solo violin, we construct a ‘phylogenetic’ tree, representing the relationship between those performances. The tree supports several interesting relations previously conjectured by the musicology community, such as the importance of date of birth and recording period in determining interpretative style. Our work also highlights some unexpected inter-connections between performers, and challenges previous assumptions regarding the significance of educational background and affiliation to the historically informed performance (HIP) style.

Preface to the special issue on "Motion analysis of musical performance."

Preface to the special issue on "Motion analysis of musical performance."

Analysing musical culture in nineteenth-century Europe: towards a musical turn?

The aim of this article is to establish the extent to which the history of music can offer new perspectives on the modern period. We need a change of perspective, moving away from the aesthetic debates on music to an investigation of actual experiences and practices of participants. Audience behaviour provides a link between musical production and society. In order to make opera houses and concert halls visible as social spheres, this article draws on examples from the musical life of Berlin and London in the 1800s. Music should no longer be regarded as a peripheral phenomenon, but instead as a potential historical question. The analysis of musical performances prompts at least one: music matters.

Analysing musical performance through functional data analysis: rhythmic structure in Schumann's Träumerei

Functional data analysis (FDA) is a relatively new branch of statistics devoted to describing and modelling data that are complete functions. Many relevant aspects of musical performance and perception can be understood and quantified as dynamic processes evolving as functions of time. In this paper, we show that FDA is a statistical methodology well suited for research into the field of quantitative musical performance analysis. To demonstrate this suitability, we consider tempo data for 28 performances of Schumann's Träumerei and analyse them by means of functional principal component analysis (one of the most powerful descriptive tools included in FDA). Specifically, we investigate the commonalities and differences between different performances regarding (expressive) timing, and we cluster similar performances together. We conclude that musical data considered as functional data reveal performance structures that might otherwise go unnoticed.

A setup for measurement of bowing parameters in bowed‐string instrument performance

An accurate measurement of instrumentalists' actions in playing situations is the basis for several research topics such as musical performance analysis, control of sound synthesis algorithms or effective validation of theoretical results in musical acoustics. We present a setup for a complete and accurate measurement of bowing parameters in bowed‐string instrument performance, with minimum interference for the player. The method is based on the combined use of motion capture and sensors attached to the bow. An optical motion capture system was used to track the position and orientation (6 DOF) of the bow and the instrument. In addition, sensors were attached to the frog of the bow for measuring bow acceleration in different axes and bow force exerted on the strings. Both sensors can be easily attached to any bow. We will describe how the data from the different systems are combined for an accurate calculation of bowing parameters, such as bow‐bridge distance, bow velocity, bow acceleration, bow angles and bow pressure. Finally, we will present examples of measurements and briefly discuss some potential applications.