Networked Music Performance Research Articles

Implementation and optimization of Burg’s method for real-time packet loss concealment in networked music performance applications

AbstractIn networked music performance (NMP) applications, which entail real-time audio streaming over the Internet, strict latency requirements are needed to ensure a realistic interaction between geographically dispersed musicians. Thus, NMP applications typically leverage uncompressed audio and unreliable transport protocols to avoid unnecessary processing and re-transmission delays. Given that no guarantee on packet delivery is offered, NMP applications must deal with late/lost audio packets to mitigate the impact of the resulting audio artifacts on the quality of the playback audio stream. This paper explores an audio packet loss concealment (PLC) technique based on autoregressive (AR) models. In particular, it investigates the algorithmic implementation of Burg’s method and the parameters configuration that offers the best trade-off between prediction error and computational time requirements. The purpose is to find the most suitable solution capable of running on a Raspberry Pi 4B within the real-time audio boundaries imposed by NMP applications. Additionally, we analyze the computational time required to fit the model and predict future samples by considering six implementations and various compilation flags. Results confirm that AR models can predict future audio samples more accurately than traditional PLC approaches, which consist of filling audio gaps with silence or repeating the last received audio segment. Furthermore, results demonstrate the effectiveness of the proposed solution in meeting the strict latency requirements when deployed on a Raspberry Pi 4B.

Ambiguous Devices as an example of a networked relational encounter

ABSTRACT This article uses the author’s collaborative environment for networked music performance, Ambiguous Devices (developed with Paul Stapleton), as a case study for illustrating a shift in thinking from a transactional to a relational understanding of networked encounters. It draws on Gill’s notions of relational interfaces and Buber’s understanding of dialogic relations as a way of understanding ways that people might communicate with others at a distance. It looks at an embodied theory of interaction that relates to intersubjective and intercorporeal ways of knowing the other and then looks at ways that these relationships can be facilitated over spatially separate sites of interaction such as those found in networked performance scenarios. Finally, it puts forward a notion of relational encounters as a way thinking about these kinds of affective interactions at a distance and discusses the designed properties of Ambiguous Devices that support and facilitate this kind of interaction.

A web-based distributed system for integrating mobile music in choral performance

This paper presents an Internet of Musical Things system designed to enhance the singing practices of conventional vocal ensembles with electronic sounds generated by smartphones. The system comprises a small loudspeaker connected to a smartphone running a web-based app that generates sounds locally to the chorister, who uses it while singing. An evaluation of the user experience was conducted through three experiments involving a small choir of 9 choristers and a conductor. In the first experiment, the system was utilized in a co-located setting, where choristers shared the same physical space. In the second experiment, the system was employed in a remote setting, where geographically displaced choristers were connected through a networked music performance system. In the third experiment, a hybrid condition was implemented where part of the choir was co-located and part was remotely connected. Overall, results show that the application can be successfully utilized to augment the practice and experience of choir singing, leading to novel forms of musical expression. We provide a critical reflection where we discuss the lessons learned, strengths, limitations, and possible future developments.

The ‘Playing Together, Apart Framework’: A framework for communication in networked music performance

The 'Playing Together, Apart Framework' is a framework for understanding communication between musicians in networked music performance (NMP). Previous research has largely focused on technical issues such as latency and synchronization, and this framework aims to extend this research into further understanding of the experiences of, and the communication between, musicians in NMP, and the factors affecting this. The framework is based on duo musicians, working in informal NMP situations, playing acoustic instruments. The framework shows the audio and video communication paths, and the influences on both the transmission and reception elements of the communication chain in NMP. General issues in NMP, such as the rhythmic content of the music, the expertise and experience of the musicians when dividing attention, the use of video and the socio-emotional and professional relationships between musicians, are examined in further detail. This framework is proposed as a basis for further research into NMP, and for expansion and development by researchers.

A Networked Hybrid Interface for Audience Sonification and Machine Learning

Lick the Toad is an ongoing project developed as a web based interface that runs in modern browsers. It provides a custom made platform to collect user data accessed from mobile devices, such as smartphones, tablets etc. The system offers a tool for interactive collective sonification supporting networked music performance. It can be used in various contexts, such as an onsite installation, or for the distribution of raw data for live coding performances making it a versatile component for an array of creative practices. Of these, live coding which is one of the author's artistic approach to create live performances is demonstrated in this article highlighting and elaborating on technical and musical aspects of this approach. Final sections outline the system as a tool for live coding performances and cover a series of potential interactions integrating audience and/or using it independently alike.

Introducing the Networked Music Performance Library

In this article we present the Networked Music Performance Library, which documents research and practice in the field of networked music performance. The library was developed in response to the skyrocketing interest in networked music due to social distancing and travel restrictions during the COVID-19 pandemic in 2020. In this article, we discuss the library’s working environment and briefly describe several similar earlier projects by other researchers. The main goals we set for the library were to collect materials, organise accessible and straightforward ways of navigating the collection, including knowledge from a wider context of time-based arts (theatre, dance and other time-based art forms), and create a community of users and contributors. We describe the main methods we use to find the works for the library and examine the current state of our collection, which has grown in the last year. Through an analysis of the entries’ metadata, we provide several examples of professionals’ use of the library. Current library users include performing musicians, teachers and composers. As the project is ongoing, we outline the scope of our future work as well as analyse several possible post-COVID applications for networked music, which include assisting socially isolated musicians, teaching, travel and new pandemics. We also show how this library could contribute to the development of new electroacoustic approaches.

Improvising Inside a House of Cards: New performance and music-making through a collective networked instrument

This article presents experiences and reflections related to performing improvised, live processed electroacoustic music within a context of networked music performance. The musical interaction is performed through a new collective networked instrument, and we report how the ensemble ‘Magnify the Sound’, consisting of two of the authors of this article, meets the instrument in different networked performance situations, and how this is related to the affordance of the instrument. In our performances the network is inherent to our artistic practice, and we experience a phenomenological and somatic transformation in our roles as musicians, from individual instrumentality to shared instrumentality. The instrument invites new forms of music-making and contributes in fundamental ways to the ensemble’s musical communication and artistic expression. In the present article we outline our methods of working artistically with the networked instrument, and we point at some artistic results. We then discuss how the collective instrument has facilitated new performance and musical practice within the network.

Benchmarking Networked Music Performance Tools With the NMP-Bench Model and Architecture

Benchmarking Networked Music Performance Tools With the NMP-Bench Model and Architecture

Temporal Coordination in Piano Duet Networked Music Performance (NMP): Interactions Between Acoustic Transmission Latency and Musical Role Asymmetries

Today’s audio, visual, and internet technologies allow people to interact despite physical distances, for casual conversation, group workouts, or musical performance. Musical ensemble performance is unique because interaction integrity critically depends on the timing between each performer’s actions and when their acoustic outcomes arrive. Acoustic transmission latency (ATL) between players is substantially longer for networked music performance (NMP) compared to traditional in-person spaces where musicians can easily adapt. Previous work has shown that longer ATLs slow the average tempo in ensemble performance, and that asymmetric co-actor roles and empathy-related traits affect coordination patterns in joint action. Thus, we are interested in how musicians collectively adapt to a given latency and how such adaptation patterns vary with their task-related and person-related asymmetries. Here, we examined how two pianists performed duets while hearing each other’s auditory outcomes with an ATL of 10, 20, or 40 ms. To test the hypotheses regarding task-related asymmetries, we designed duets such that pianists had: (1) a starting or joining role and (2) a similar or dissimilar musical part compared to their co-performer, with respect to pitch range and melodic contour. Results replicated previous clapping-duet findings showing that longer ATLs are associated with greater temporal asynchrony between partners and increased average tempo slowing. While co-performer asynchronies were not affected by performer role or part similarity, at the longer ATLs starting performers displayed slower tempos and smaller tempo variability than joining performers. This asymmetry of stability vs. flexibility between starters and joiners may sustain coordination, consistent with recent joint action findings. Our data also suggest that relative independence in musical parts may mitigate ATL-related challenges. Additionally, there may be a relationship between co-performer differences in empathy-related personality traits such as locus of control and coordination during performance under the influence of ATL. Incorporating the emergent coordinative dynamics between performers could help further innovation of music technologies and composition techniques for NMP.

Aesthetic and technical strategies for networked music performance.

Networked music is no longer a future genre: the global quarantine event of 2020 launched the concept of performing together over the Internet into the mainstream. While the demand for performing at a distance may be a new imperative, musicians find themselves faced with technological and performative processes that do not appear to be suitable for performing music together online, due particularly to network latency which disrupts the ability for musicians to synchronize. The research presented in this paper investigates and challenges the reasons why networked music is not readily embraced by musicians and describes how that might change, by way of interviews with practitioners and an in-depth review of the technical constraints. Limitations that might cause frustration are in fact shown to have creative strategies that give rise to aesthetic approaches, distinct to the platform. By exploiting the constraints, in tandem with developing technology designed specifically for remote performance, aesthetic implications arise that not only accommodate the inconveniences of latency and acoustic feedback but can help us adapt and transform how we engage in real-time online, towards a future where we can imagine performing together over even more dramatic distances such as high-latency, low-bandwidth locations outside of urban areas—or even over galactic distances.

A Presence- and Performance-Driven Framework to Investigate Interactive Networked Music Learning Scenarios

Cooperative music making in networked environments has been subject of extensive research, scientific and artistic. Networked music performance (NMP) is attracting renewed interest thanks to the growing availability of effective technology and tools for computer-based communications, especially in the area of distance and blended learning applications. We propose a conceptual framework for NMP research and design in the context of classical chamber music practice and learning: presence-related constructs and objective quality metrics are used to problematize and systematize the many factors affecting the experience of studying and practicing music in a networked environment. To this end, a preliminary NMP experiment on the effect of latency on chamber music duos experience and quality of the performance is introduced. The degree of involvement, perceived coherence, and immersion of the NMP environment are here combined with measures on the networked performance, including tempo trends and misalignments from the shared score. Early results on the impact of temporal factors on NMP musical interaction are outlined, and their methodological implications for the design of pedagogical applications are discussed.

Playing Together, Apart

Classical musicians in ensembles are accustomed to performing in the same physical space. However, situations such as networked music performance (NMP) require physical separation with audio and sometimes video links. Effects of latency on synchronization have been extensively studied; however, research is limited on the effects of physical separation on the subjective experience of musicians. This separation is likely to have an effect on interaction between musicians as usual channels of communication are interrupted. The impact of this physical separation on the experience of classical musicians in a woodwind soloist and piano accompanist setting was investigated. Three pairs of musicians were recorded in acoustically isolated spaces with audio and video links, and were then interviewed using semi-structured interview techniques. Five themes emerged from the data, namely: adaptability, communication, performance, impact on the musicians, and relationships. Within these themes, musical issues, communication, and social interactions were found to be most challenging for separated musicians, while adaptability helped the musicians in this situation. The video link was used rarely when playing. These issues are important in NMP and are related to the physical separation of the musicians, rather than problems such as latency, which are well documented.

Improving networked music performance systems using application‐network collaboration

SummaryNetworked Music Performance (NMP) systems involve musicians located in different places who perform music while staying synchronized via the Internet. The maximum end‐to‐end delay in NMP applications is called Ensemble Performance Threshold (EPT) and should be less than 25 milliseconds. Due to this constraint, NMPs require ultra–low‐delay solutions for audio coding, network transmission, relaying, and decoding, each one a challenging task on its own. There are two directions for study in the related work referring to the NMP systems. From the audio perspective, researchers experiment on low‐delay encoders and transmission patterns, aiming to reduce the processing delay of the audio transmission, but they ignore the network performance. On the other hand, network‐oriented researchers try to reduce the network delay, which contributes to reduced end‐to‐end delay. In our proposed approach, we introduce an integration of dynamic audio and network configuration to satisfy the EPT constraint. The basic idea is that, the major components participating in an NMP system, the application and the network interact during the live music performance. As the network delay increases, the network tries to equalize it by modifying the routing behavior using Software‐Defined Networking principles. If the network delay exceeds a maximum affordable threshold, the network reacts by informing the application to change the audio processing pattern to overcome the delay increase, resulting in below EPT end‐to‐end delay. A full prototype of the proposed system was implemented and extensively evaluated in an emulated environment.

Networks for Future Services in a Smart City: Lessons Learned from the Connected OFCity Challenge 2017

The drive toward ubiquitous communications has long been encompassed by the concept of a connected or smart city. The idea that data transfer and real-time data analysis can enhance the quality of life for urban inhabitants is compelling, and one can easily envision the provision of exciting new services and applications that such an information- driven city could provide. The challenge in achieving a truly smart city stems largely from communications technologies -- fixed line, wireless, backhaul, and fronthaul -- and how these are combined to provide fast, reliable, and secure communications coverage. Here, we report on the key observations from the Connected OFCity Challenge competition, held at OFC 2017, which addressed the fixed and wireless access network requirements for smart cities. It is shown that from a technological perspective, future optical networks will be capable of securely supporting extremely low-latency and high-bandwidth applications. However, as shown by using Networked Music Performance as a particularly challenging example application, how readily this is achieved will depend on the interplay between wired and wireless access services.

Anticipatory Networked Communications for Live Musical Interactions of Acoustic Instruments

Performing music over computer networks requires highly reliable infrastructures with respect to latency elimination, bandwidth availability and error alleviation. Hence, commonly available networks or large communication distance cannot effectively cope with real-time, high-fidelity and uninterrupted musical communications. An intuitive solution to this problem can be offered by employing mechanisms for musical prediction and anticipation which can help mitigate technical impediments. The idea is that knowing in advance the evolution of a music piece can offer effective reduction of communication latencies and data rates. Furthermore, musical anticipation can be used to eliminate errors owing to data loss in the network transmission path. We present the development of a software prototype that uses anticipatory processes to support networked music performance. The adopted methodology is inspired by computer accompaniment systems, in which a software agent follows the performance of a human musician in real-time, by synchronizing a pre-recorded musical accompaniment to the expressive nuances of live interpretation. The paper presents an overview of relevant research initiatives, the computational approach implemented for anticipatory networked performances and an experimental validation that demonstrates current progress and reveals constraints and challenges pending further investigation.

An Overview on Networked Music Performance Technologies

Networked music performance (NMP) is a potential game changer among Internet applications, as it aims at revolutionizing the traditional concept of musical interaction by enabling remote musicians to interact and perform together through a telecommunication network. Ensuring realistic performance conditions, however, constitutes a significant engineering challenge due to the extremely strict requirements in terms of network delay and audio quality, which are needed to maintain a stable tempo, a satisfying synchronicity between performers and, more generally, a high-quality interaction experience. In this paper, we offer a review of the psycho-perceptual studies conducted in the past decade, aimed at identifying latency tolerance thresholds for synchronous real-time musical performance. We also provide an overview of hardware/software enabling technologies for NMP, with a particular emphasis on system architecture paradigms, networking configurations, and applications to real use cases.

Multi-modal musical environments for mixed-reality performance

This article describes a series of multi-modal networked musical performance environments designed and implemented for concert presentation at the Torino-Milano (MiTo) Festival (Settembre musica, 2009, http://www.mitosettembremusica.it/en/home.html) between 2009 and 2010. Musical works, controlled by motion and gestures generated by in-engine performer avatars will be discussed with specific consideration given to the multi-modal presentation of mixed-reality works, combining both software-based and real-world traditional musical instruments.

The Effects of Network Delay on Tempo in Musical Performance

Internet-based collaborative networking applications, such as instant messaging, voice-over-IP telephony, and social networking, have displaced traditional communication services and redefined social interaction. The Internet has also transformed the music industry, revolutionizing the way music is distributed and marketed. Yet despite these two powerful trends, the intersection—where collaboration and music meet in online musicmaking—has remained merely a curiosity. Why? Artistically pleasing online audio collaboration requires network delay less than that encountered typically in the Internet. The bandwidth required for high-quality audio exceeds the bandwidth that is generally available on consumer-oriented broadband access (cable and digital subscriber line [DSL]) systems. The emergence of Web 2.0, broadly defined as Web-based communities such as social-networking sites that facilitate sharing of ideas among Web users, has been significant for many existing online communities. One such community, made up of real-time Web-based music collaborators using systems for networked musical performance (NMP), is in its infancy. An online NMP application lets musicians from across the globe play together over the Internet, as if they were together in a studio. With online music-making (either improvisatory or strictly notated), musicians can create ensembles without location bounds, searching for musicians around the world. The quality of the user’s experience is critical to the success of this Web application. However, because performing artists are highly sensitive to delay, network latency affects the quality of the user experience of online music-making. To achieve a good user experience the latency over the network has to be within reasonable bounds. If the delay is excessive, then the musicians will not be

Networked Music & Soundart Timeline (NMSAT) Excerpts of Part One: Ancient and Modern History, Anticipatory Literature, and Technical Developments References

This article presents a sample of references present in the existing Networked Music & SoundArt Timeline (NMSAT) database. The NMSAT is a research project led and initiated in 2008 by Jérôme Joy within the Locus Sonus lab. It is conceived as a monitor and a pool of informations on history of networked music and sound (Joy & Sinclair, 2009). It aims to provide an overview of practices and techniques in the realm of networked music and networked sonic performance from ancient history to the present (2008), related to sound transmission and distance listening. It consists of a collection of references from various online and bibliographical documents, articles, workshop notes and so on. The NMSAT offers a valuable resource made available to actors in the artistic and scientific spheres. Because it is not possible to summarise in one article the large spectrum of references from all periods, we decided to present a sample related to Part 1 of the database (Part 2 concerns references of networked music and soundart, technological developments and contemporary history; Part 3 contains a bibliography of reference papers). This sample lists a selection of references covering the period from the sixteenth century to the middle of the nineteenth century, just before the major developments of the telephone by Alexander Graham Bell and Elisha Gray from 1876. Situated before the realisation of apparatuses of telecommunication, the selected period can reveal, across the interactions between inventions, ‘uchronias’ (alternate history) and anticipation (Apollinaire, 1916; see also Valéry, 1960 [1928]), the paradigm of listening, and of manufacturing listenings, involved in the systems of sound transmission (and transport) and sound actions at a distance. This will highlight the contextualisation and foster analysis of the development of audio networked practices towards an ‘organology’ (the science of musical instruments and their classification, and more largely, the science of organs or of anything considered as an organic structure) of such of systems and the exploration of the soniferous condition of electronic networks (interconnected spaces, audiences and ‘streamers’) (Joy, 2009). Of course, all the other periods, before and after these dates, are covered by the existing database currently developed by the NMSAT group. Currently this database comprises more than 1400 pages. This selection illustrates the historical and theoretical context of networked music and soundart, and more largely of network art. The database is maintained by the NMSAT Development Committee. Members (in alphabetical order and in September 2009) are: Brett Ian Balogh, Álvaro Barbosa, Clarisse Bardiot, Dinahbird, Samuel Bordreuil, Andrea Cera, Julien Clauss, Jean Cristofol, John Levack Drever, Alejo Duque, Björn Eriksson, Scott Fitzgerald, Golo Föllmer, Jean-Paul Fourmentraux, Peter Gena, Gh Hovagymian, Jérôme Joy, Rahma Khazam, Brandon Labelle, Anne Laforet, Marie Lechner, Eric Leonardson, Patrice Loubier, Gilles Malatray, Cédric Maridet, Sylvain Marquis, Luc Martinez, Andra McCartney, Udo Noll, Pauline Oliveros, Julien Ottavi, Jenny Pickett, Andrea Polli, Céline Prunneaux, Pedro Rebelo, Marc Relieu, Alain Renaud, Jean-Philippe Renoult, Annick Rivoire, Jocelyn Robert, Anne Roquigny, Peter Sinclair, Douglas E. Stanley, Atau Tanaka, Dante Tanzi, J. Milo Taylor, Jean-Paul Thibaud, Allen S. Weiss, Yang Yeung, Laura Zattra (http://locusonus.org/nmsat).

Effects of delayed auditory feedback upon articulation.

Effects of delayed auditory feedback upon articulation.