Musical Input Research Articles

The geometry of interpersonal synchrony in human dance

Collective synchronized behavior has powerful social-communicative functions observed across several animal taxa.1,2,3,4,5,6,7 Operationally, synchronized behavior can be explained by individuals responding to shared external cues (e.g., light, sound, or food) as well as by inter-individual adaptation.3,8,9,10,11 We contrasted these accounts in the context of a universal human practice-collective dance-by recording full-body kinematics from dyads of laypersons freely dancing to music in a "silent disco" setting. We orthogonally manipulated musical input (whether participants were dancing to the same, synchronous music) and visual contact (whether participants could see their dancing partner). Using a data-driven method, we decomposed full-body kinematics of 70 participants into 15 principal movement patterns, reminiscent of common dance moves, explaining over 95% of kinematic variance. We find that both music and partners drive synchrony, but through distinct dance moves. This leads to distinct kinds of synchrony that occur in parallel by virtue of a geometric organization: anteroposterior movements such as head bobs synchronize through music, while hand gestures and full-body lateral movements synchronize through visual contact. One specific dance move-vertical bounce-emerged as a supramodal pacesetter of coordination, synchronizing through both music and visual contact, and at the pace of the musical beat. These findings reveal that synchrony in human dance is independently supported by shared musical input and inter-individual adaptation. The independence between these drivers of synchrony hinges on a geometric organization, enabling dancers to synchronize to music and partners simultaneously by allocating distinct synchronies to distinct spatial axes and body parts.

Comparison of speech and music input in North American infants' home environment over the first 2 years of life.

Infants are immersed in a world of sounds from the moment their auditory system becomes functional, and experience with the auditory world shapes how their brain processes sounds in their environment. Across cultures, speech and music are two dominant auditory signals in infants' daily lives. Decades of research have repeatedly shown that both quantity and quality of speech input play critical roles in infant language development. Less is known about the music input infants receive in their environment. This study is the first to compare music input to speech input across infancy by analyzing a longitudinal dataset of daylong audio recordings collected in English-learning infants' home environments, at 6, 10, 14, 18, and 24 months of age. Using a crowdsourcing approach, 643 naïve listeners annotated 12,000 short snippets (10 s) randomly sampled from the recordings using Zooniverse, an online citizen-science platform. Results show that infants overall receive significantly more speech input than music input and the gap widens as the infants get older. At every age point, infants were exposed to more music from an electronic device than an in-person source; this pattern was reversed for speech. The percentage of input intended for infants remained the same over time for music while that percentage significantly increased for speech. We propose possible explanations for the limited music input compared to speech input observed in the present (North American) dataset and discuss future directions. We also discuss the opportunities and caveats in using a crowdsourcing approach to analyze large audio datasets. A video abstract of this article can be viewed at https://youtu.be/lFj_sEaBMN4 RESEARCH HIGHLIGHTS: This study is the first to compare music input to speech input in infants' natural home environment across infancy. We utilized a crowdsourcing approach to annotate a longitudinal dataset of daylong audio recordings collected in North American home environments. Our main results show that infants overall receive significantly more speech input than music input. This gap widens as the infants get older. Our results also showed that the music input was largely from electronic devices and not intended for the infants, a pattern opposite to speech input.

Construction and Implementation of Content-Based National Music Retrieval Model Under Deep Learning

This research mainly studies the construction and implementation of the content-based folk music retrieval model. Firstly, it studies the music automatic annotation method based on deep learning, and then proposes the tag conditional random field music automatic annotation method, and then constructs the music annotation depth neural network model combining a variety of music representation and attention mechanism. Finally, it analyzes the proposed folk music retrieval model the effectiveness of the cable model is verified and its performance is evaluated. The results show that in Glu module, Glu blocks had better performance in music annotation, and the music annotation results of each index in music hierarchical sequence modeling are better, which ensures the effectiveness of music annotation. Compared with other algorithms, the AUC tag score of the proposed method is the highest, which is 0.913; it can better model the mapping relationship between the audio features of music input to the text tag and has higher scores on all evaluation indicators.

Multidisciplinary Improvisation Consort in Brasov

Interacting electronic processing and spatialization with acoustic instruments has been done by many performers over the past years. This consort aims to continue the exploration of musicianship through technological interaction along with visual music inputs by means of video artistry collaboration. We thrive in improvising and reacting to each other by what is presented in terms of sound and image, where ultimately a musical outcome is presented to the audience.

Facilitating Intercultural Communicative Competence through a Multiliteracies approach: musical input as the missing link

Foreign Language (FL) learning has undergone a shift away from the communicative approach and towards a pedagogy of Multiliteracies. In an ever-changing society characterized by a digital and multicultural context, the necessity of developing Intercultural Communicative Competence (ICC) in FL learning across all levels through a pedagogy of Multiliteracies has become a priority. In this regard, musical input is an ideal medium to explore culture in the FL learning classroom which enables the integration of affective experiences from an intercultural perspective. The reasons for the implementation of musical input into FL learning curricula to develop ICC is twofold: on the one hand, it may serve to stimulate students’ awareness towards the target culture; and, on the other hand, it may be an effective way to engage students in intercultural encounters from an affective point of view. A corpus has been designed in order to analyze those studies on the development of ICC in FL learning across all levels through a Multiliteracies approach. The findings demonstrate that the use of musical input to develop ICC in the FL learning classroom through a pedagogy of Multiliteracies is scarce. Moreover, this study seeks to promote musical input-based activities to facilitate the development of ICC in the FL learning classroom through a Multiliteracies pedagogy.

Multi-dimensional fusion: transformer and GANs-based multimodal audiovisual perception robot for musical performance art.

In the context of evolving societal preferences for deeper emotional connections in art, this paper explores the emergence of multimodal robot music performance art. It investigates the fusion of music and motion in robot performances to enhance expressiveness and emotional impact. The study employs Transformer models to combine audio and video signals, enabling robots to better understand music's rhythm, melody, and emotional content. Generative Adversarial Networks (GANs) are utilized to create lifelike visual performances synchronized with music, bridging auditory and visual perception. Multimodal reinforcement learning is employed to achieve harmonious alignment between sound and motion. The study leverages Transformer models to process audio and video signals in robot performances. Generative Adversarial Networks are employed to generate visually appealing performances that align with the musical input. Multimodal reinforcement learning is used to synchronize robot actions with music. Diverse music styles and emotions are considered in the experiments. Performance evaluation metrics include accuracy, recall rate, and F1 score. The proposed approach yields promising results across various music styles and emotional contexts. Performance smoothness scores exceed 94 points, demonstrating the fluidity of robot actions. An accuracy rate of 95% highlights the precision of the system in aligning robot actions with music. Notably, there is a substantial 33% enhancement in performance recall rate compared to baseline modules. The collective improvement in F1 score emphasizes the advantages of the proposed approach in the realm of robot music performance art. The study's findings demonstrate the potential of multimodal robot music performance art in achieving heightened emotional impact. By combining audio and visual cues, robots can better interpret and respond to music, resulting in smoother and more precise performances. The substantial improvement in recall rate suggests that the proposed approach enhances the robots' ability to accurately mirror the emotional nuances of the music. These results signify the potential of this approach to transform the landscape of artistic expression through robotics, opening new avenues for emotionally resonant performances.

Creating musical features using multi-faceted, multi-task encoders based on transformers

Computational machine intelligence approaches have enabled a variety of music-centric technologies in support of creating, sharing and interacting with music content. A strong performance on specific downstream application tasks, such as music genre detection and music emotion recognition, is paramount to ensuring broad capabilities for computational music understanding and Music Information Retrieval. Traditional approaches have relied on supervised learning to train models to support these music-related tasks. However, such approaches require copious annotated data and still may only provide insight into one view of music—namely, that related to the specific task at hand. We present a new model for generating audio-musical features that support music understanding, leveraging self-supervision and cross-domain learning. After pre-training using masked reconstruction of musical input features using self-attention bidirectional transformers, output representations are fine-tuned using several downstream music understanding tasks. Results show that the features generated by our multi-faceted, multi-task, music transformer model, which we call M3BERT, tend to outperform other audio and music embeddings on several diverse music-related tasks, indicating the potential of self-supervised and semi-supervised learning approaches toward a more generalized and robust computational approach to modeling music. Our work can offer a starting point for many music-related modeling tasks, with potential applications in learning deep representations and enabling robust technology applications.

John Eccles’s Collection of Songs (1704) and John Walsh’s Publishing Model in the Early Eighteenth Century

Reviewing Eric Harbeson’s edition of John Eccles’s Judgment of Paris (1702)—part of the ongoing Works of John Eccles published by A-R Editions—Bryan White recently called for a careful appraisal of the composer’s anthology A Collection of Songs for One Two and Three Voices (1704) in order to understand better its relationship with other sources. This article takes up White’s challenge through close analysis of the 1704 Songs, focusing on the songs Walsh had published prior to Eccles’s collection. It is this aspect that makes the Songs fundamentally different from Purcell’s Orpheus Britannicus (1698) and Blow’s Amphion Anglicus (1700), both of which were typeset ex nihilo. By contrast, about one-third of the 1704 Songs not only had been previously published by Walsh but were also reprinted from the very same plates, although with frequent corrections, compositional revisions, and standardized peritextual materials (page numbering, titles and attributions, etc.). As such, the Songs offers a unique window into the editorial process in the latter stages of its preparation. After detailing the publication history of the Songs, I show that Eccles must have taken a proactive role in preparing the publication, and I examine the evidence for his musical input. The conclusions drawn from this analysis have broader implications for Walsh’s working methods, in particular the relationship between his separate songsheets of ca. 1700 and the composite publications that increasingly dominated his catalog in the first two decades of the new century.

A Music-Driven Deep Generative Adversarial Model for Guzheng Playing Animation.

To date relatively few efforts have been made on the automatic generation of musical instrument playing animations. This problem is challenging due to the intrinsically complex, temporal relationship between music and human motion as well as the lacking of high quality music-playing motion datasets. In this article, we propose a fully automatic, deep learning based framework to synthesize realistic upper body animations based on novel guzheng music input. Specifically, based on a recorded audiovisual motion capture dataset, we delicately design a generative adversarial network (GAN) based approach to capture the temporal relationship between the music and the human motion data. In this process, data augmentation is employed to improve the generalization of our approach to handle a variety of guzheng music inputs. Through extensive objective and subjective experiments, we show that our method can generate visually plausible guzheng-playing animations that are well synchronized with the input guzheng music, and it can significantly outperform the state-of-the-art methods. In addition, through an ablation study, we validate the contributions of the carefully-designed modules in our framework.

Feasibility of Using Bande à Part to Aid French Language Learners

This pilot study examines users’ perceptions of Bande à Part, a music application designed for learners of French. The technology acceptance model (TAM) was adopted to investigate users’ perceptions of the app’s usability and potential for second language (L2) learning. The model’s two constructs, perceived usefulness and perceived ease of use, and one added factor, perceived enjoyment, formed the main predictors of users’ intentions to continue using the app. Mean scores for the predictors were: perceived usefulness = 4.27/6, perceived ease of use = 3.88/6, and perceived enjoyment = 3.95/6, which are confirmed by the survey results that show that 10 of 13 participants intend to continue using the app. Qualitative results suggest that the app enhances users’ ability to notice targeted forms in the musical input (e.g., liaison, gender) and, corroborating the quantitative data, suggest that users find the features in the app useful. Several comments also indicate that the ease of use could be improved (e.g., improved mobile device access). This study helps to establish the TAM in Computer-Assisted Language Learning (CALL) literature and forms the basis for future work evaluating how songs aid L2 acquisition.

Using a Developmental-Ecological Approach to Understand the Relation Between Language and Music.

Neurocognitive and genetic approaches have made progress in understanding language-music interaction in the adult brain. Although there is broad agreement that learning processes affect how we represent, comprehend, and produce language and music, there is little understanding of the content and dynamics of the early language-music environment in the first years of life. A developmental-ecological approach sees learning and development as fundamentally embedded in a child’s environment, and thus requires researchers to move outside of the lab to understand what children are seeing, hearing, and doing in their daily lives. In this paper, after first reviewing the limitations of traditional developmental approaches to understanding language-music interaction, we describe how a developmental-ecological approach can not only inform developmental theories of language-music learning, but also address challenges inherent to neurocognitive and genetic approaches. We then make suggestions for how researchers can best use the developmental-ecological approach to understand the similarities, differences, and co-occurrences in early music and language input.

Heritage

Heritage

Interactive Musical Partner: A Demonstration of Musical Personality Settings for Influencing the Behavior of an Interactive Musical Generation System

The Interactive Musical Partner (IMP) is software designed for use in duo improvisations, with one human improviser and one instance of IMP, focusing on a freely improvised duo aesthetic. IMP has Musical Personality Settings (MPS) that can be set prior to performance, and these MPS guide the way IMP responds to musical input. The MPS also govern the probability of particular outcomes from IMP's creative algorithms. The IMP uses audio data feature extraction methods to listen to the human partner, and react to, or ignore, the human’s musical input, based on the current MPS. This demonstration shows how the MPS interface with IMP's generative algorithm.

Predicting the Preference for Sad Music: The Role of Gender, Personality, and Audio Features

The “tragedy paradox” of music, avoiding experiencing negative emotions but enjoying the sadness portrayed in music, has attracted a great deal of academic attention in recent decades. Combining experimental psychology research methods and machine learning techniques, this study (a) investigated the effects of gender and Big Five personality factors on the preference for sad music in the Chinese social environment and (b) constructed sad music preference prediction models using audio features and individual features as inputs. Statistical analysis found that males have a greater preference for sad music than females do, and that gender and the extraversion factor are involved in significant two-way interactions. The best-performing random forest regression shows a low predictive effect on the preference for sad music ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$R^{2} =0.138$ </tex-math></inline-formula> ), providing references for music recommendation systems. Finally, the importance-based model interpretation feature reveals that, in addition to the same music inputs (audio features), the perceived relaxation and happiness of music play an important role in the prediction of sad music preferences.

Real-Time Routing Protocol for Music Application With Multiple Outputs

We present a client-server based smartphone application that act as a music input device with several outputs. The application allows user to interact with one phone as the input and the server and hear the sound from several connected smartphones that act as clients. Since transmitting data may require some time, we need to deal with transmission delay. Hence, we developed an algorithm to detect the delay and synchronize these values between all connected devices. By agreeing to a single delay value, we were able to minimize the difference in playtime between server and connected clients. We performed the test by building an Android-based smartphone application, installing it to multiple devices, and connecting them to a WiFi-based local area network. The test shows that our proposed system has successfully reduce the difference between devices and the connected devices were able to produce sounds with insignificant delay.

Input musical fluidez lectora en el desarrollo de la expresión escrita en estudiantes adultos de español como L2

Music shares characteristics with language. The link between both and their impact on second language learning has been studied over the years. In this paper we focus on studying the development of writing and its relation with two variables: reading fluency and musical input. This experimental study fits within an action-research framework. The sample used on this project consisted of 13 students of Spanish with a B1 level of language proficiency. The participants were divided into two groups, one of them received musical stimulation while the other did not. Additionally, both groups were given a writing test, based on an activity created by Mario Rinvolucri, along with an oral reading fluency test. The correlational study points out that there may be a positive effect on the use of music as input added to the written text.

Musician and Nonmusician Hearing Aid Setting Preferences for Music and Speech Stimuli.

Purpose The purpose of this study was to evaluate potential group differences between musicians and nonmusicians in their self-adjusted (SA) gain and compression settings for both music and speech stimuli. Speech recognition, sound quality, and strength of preference for the SA settings and the original prescriptive (National Acoustic Laboratories-Nonlinear 2 [NAL-NL2]) settings were also compared. Method Participants included 12 musician ( M = 60 years) and 12 nonmusician ( M = 55 years) adult hearing aid users with mild-moderate hearing loss, on average. Self-adjustments were made to hearing aid gain and compression settings for 2 music stimuli and a speech stimulus. Speech recognition in quiet and noise, sound quality for 6 dimensions (clarity, pleasantness, naturalness, fullness, brightness, and overall impression), and strength of preference ratings using paired comparisons were then assessed at both the NAL-NL2 settings and the participants' SA settings. Results On average, self-adjustments made by both groups were quite small (< 5 dB for gain and < 0.5 for compression ratio). Furthermore, SA changes to gain and compression ratio were not significantly different for musicians versus nonmusicians or for music versus speech. Finally, speech perception performance and sound quality ratings did not differ for the SA settings versus the NAL-NL2 settings, with the exception of the naturalness sound quality dimension. Conclusions These data suggest that a gain-frequency response specific to musicians and/or music inputs may not be necessary. Thus, current, validated prescriptive methods continue to be well supported as an appropriate starting place for listeners with mild-moderate hearing loss using open hearing aid fittings.

Composing with Biomemristors: Is Biocomputing the New Technology of Computer Music?

Abstract Our research concerns the development of biocomputers using electronic components grown out of biological material. This article reports the development of a novel biological memristor and an approach to using such biomemristors to build interactive generative music systems. The memristor is an electronic component regarded as a resistor with memory. After an introduction to harnessing the Physarum polycephalum organism to implement biomemristors, the article presents PhyBox, a biocomputer that uses four biomemristors to generate music interactively. The resistance of a biomemristor varies as a function of the voltage that has passed through it. Music input is represented in terms of voltage transitions and music output is encoded as measurements of the current yielded by the system's memristive behavior. An example of a musical composition using PhyBox is detailed. The article concludes with a short discussion of how the combination between artificial machines and biological organisms is paving the way for the development of new technologies for music based on living processors.

Generating Time: Rhythmic Perception, Prediction and Production with Recurrent Neural Networks

In the quest for a convincing musical agent that performs in real time alongside human performers, the issues surrounding expressively timed rhythm must be addressed. Current beat tracking methods are not sufficient to follow rhythms automatically when dealing with varying tempo and expressive timing. In the generation of rhythm, some existing interactive systems ignore the pulse entirely, or fix a tempo after some time spent listening to input. Since music unfolds in time, we take the view that musical timing needs to be at the core of a music generation system. Our research explores a connectionist machine learning approach to expressive rhythm generation, based on cognitive and neurological models. Two neural network models are combined within one integrated system. A Gradient Frequency Neural Network (GFNN) models the perception of periodicities by resonating nonlinearly with the musical input, creating a hierarchy of strong and weak oscillations that relate to the metrical structure. A Long Short-term Memory Recurrent Neural Network (LSTM) models longer-term temporal relations based on the GFNN output. The output of the system is a prediction of when in time the next rhythmic event is likely to occur. These predictions can be used to produce new rhythms, forming a generative model. We have trained the system on a dataset of expressively performed piano solos and evaluated its ability to accurately predict rhythmic events. Based on the encouraging results, we conclude that the GFNN-LSTM model has great potential to add the ability to follow and generate expressive rhythmic structures to real-time interactive systems

Deep Music: Towards Musical Dialogue

Computer dialogue systems are designed with the intention of supporting meaningful interactions with humans. Common modes of communication include speech, text, and physical gestures. In this work we explore a communication paradigm in which the input and output channels consist of music. Specifically, we examine the musical interaction scenario of call and response. We present a system that utilizes a deep autoencoder to learn semantic embeddings of musical input. The system learns to transform these embeddings in a manner such that reconstructing from these transformation vectors produces appropriate musical responses. In order to generate a response the system employs a combination of generation and unit selection. Selection is based on a nearest neighbor search within the embedding space and for real-time application the search space is pruned using vector quantization. The live demo consists of a person playing a midi keyboard and the computer generating a response that is played through a loudspeaker.