Musical Interface Research Articles

A Human-Computer Interaction Strategy for An FPGA Platform Boosted Integrated "Perception-Memory" System Based on Electronic Tattoos and Memristors.

The integrated "perception-memory" system is receiving increasing attention due to its crucial applications in humanoid robots, as well as in the simulation of the human retina and brain. Here, a Field Programmable Gate Array (FPGA) platform-boosted system that enables the sensing, recognition, and memory for human-computer interaction is reported by the combination of ultra-thin Ag/Al/Paster-based electronic tattoos (AAP) and Tantalum Oxide/Indium Gallium Zinc Oxide (Ta2O5/IGZO)-based memristors. Notably, the AAP demonstrates exceptional capabilities in accommodating the strain caused by skin deformation, thanks to its unique structural design, which ensures a secure fit to the skin and enables the prolonged monitoring of physiological signals. By utilizing Ta2O5/IGZO as the functional layer, a high switching ratio is conferred to the memristor, and an integrated system for sensing, distinguishing, storing, and controlling the machine hand of multiple human physiological signals is constructed together with the AAP. Further, the proposed system implements emergency calls and smart homes using facial electromyogram signals and utilizing logical entailment to realize the control of the music interface. This innovative "perception-memory" integrated system not only serves the disabled, enhancing human-computer interaction but also provides an alternative avenue to enhance the quality of life and autonomy of individuals with disabilities.

Rethinking Instrumental Interface Design: The MetaBow

Abstract Although extensive research has been conducted over the past two decades on the development and implementation of sensors using inertial movement units for violin bows, most of these new interfaces share a common drawback—bulky and user-unfriendly design. Despite the advancements in sensor data processing for composition, performance, and pedagogy, interface design remains a critical bottleneck for the real-world implementation of these sensor-embedded devices. Our study introduces MetaBow, a low-cost, nonintrusive frog design for violin bows that can accommodate either a standard sensor kit or a custom-designed board. This interface eliminates the need for additional physical training and maintains the integrity of traditional violin performance mechanisms. We thus view MetaBow as heralding a new era in digital music interface design for the violin family with the potential for seamless human-computer and human-machine collaboration in music practice and performance.

Perceptual Relevance of Haptic Feedback during Virtual Plucking, Bowing and Rubbing of Physically-Based Musical Resonators

The physics-based design and realization of a digital musical interface asks for the modeling and implementation of the contact-point interaction with the performer. Musical instruments always include a resonator that converts the input energy into sound, meanwhile feeding part of it back to the performer through the same point. Specifically during plucking or bowing interactions, musicians receive a handful of information from the force feedback and vibrations coming from the contact points. This paper focuses on the design and realization of digital music interfaces realizing two physical interactions along with a musically unconventional one, rubbing, rarely encountered in assimilable forms across the centuries on a few instruments. Therefore, it aims to highlight the significance of haptic rendering in improving quality during a musical experience as opposed to interfaces provided with a passive contact point. Current challenges are posed by the specific requirements of the haptic device, as well as the computational effort needed for realizing such interactions without occurrence during the performance of typical digital artifacts such as latency and model instability. Both are however seemingly transitory due to the constant evolution of computer systems for virtual reality and the progressive popularization of haptic interfaces in the sonic interaction design community. In summary, our results speak in favor of adopting nowadays haptic technologies as an essential component for digital musical interfaces affording point-wise contact interactions in the personal performance space.

Music Segmentation and Similarity Estimation Applied to a Gaze-Controlled Musical Interface

Assistive technology, especially gaze-controlled, can promote accessibility, health care, well-being and inclusion for impaired people, including musical activities that can be supported by interfaces controlled using eye tracking. Also, the Internet growth has allowed access to a huge digital music database, which can contribute to a new form of music creation. In this paper, we propose the application of Music Information Retrieval techniques for music segmentation and similarity identification, aiming at the development of a new form of musical creation using an automatic process and the optimization algorithm Harmony Search to combine segments. These techniques for segmentation and similarity of segments were implemented in an assistive musical interface controlled by eye movement to support musical creation and well-being. The experimental results can be found in [https://bit.ly/2Zl7KSC].

Assessment of Different Feature Extraction Methods for Discriminating Expressed Emotions during Music Performance towards BCMI Application.

A Brain-Computer Music Interface (BCMI) system may be designed to harness electroencephalography (EEG) signals for control over musical outputs in the context of emotionally expressive performance. To develop a real-time BCMI system, accurate and computationally efficient emotional biomarkers should first be identified. In the current study, we evaluated the ability of various features to discriminate between emotions expressed during music performance with the aim of developing a BCMI system. EEG data was recorded while subjects performed simple piano music with contrasting emotional cues and rated their success in communicating the intended emotion. Power spectra and connectivity features (Magnitude Square Coherence (MSC) and Granger Causality (GC)) were extracted from the signals. Two different approaches of feature selection were used to assess the contribution of neutral baselines in detection accuracies; 1- utilizing the baselines to normalize the features, 2- not taking them into account (non-normalized features). Finally, the Support Vector Machine (SVM) has been used to evaluate and compare the capability of various features for emotion detection. Best detection accuracies were obtained from the non-normalized MSC-based features equal to 85.57 ± 2.34, 84.93 ± 1.67, and 87.16 ± 0.55 for arousal, valence, and emotional conditions respectively, while the power-based features had the lowest accuracies. Both connectivity features show acceptable accuracy while requiring short processing time and thus are potential candidates for the development of a real-time BCMI system.

Exploring Spatio-spectral Electroencephalogram Modulations of Imbuing Emotional Intent during Active Piano Playing.

Imbuing emotional intent serves as a crucial modulator of music improvisation during active musical instrument playing. However, most improvisation-related neural endeavors have been gained without considering the emotional context. This study attempted to exploit reproducible spatio-spectral electroencephalogram (EEG) oscillations of emotional intent using a data-driven independent component analysis framework in an ecological multiday piano playing experiment. Through the four-day 32-ch EEG dataset of 10 professional players, we showed that EEG patterns were substantially affected by both intra- and inter-individual variability underlying the emotional intent of the dichotomized valence (positive vs. negative) and arousal (high vs. low) categories. Less than half (3-4) of the 10 participants analogously exhibited day-reproducible (≥ three days) spectral modulations at the right frontal beta in response to the valence contrast as well as the frontal central gamma and the superior parietal alpha to the arousal counterpart. In particular, frontal engagement facilitated a better understanding of the frontal cortex (e.g., dorsolateral prefrontal cortex and anterior cingulate cortex) and its role in intervening emotional processes and expressing spectral signatures that are relatively resistant to natural EEG variability. Such ecologically vivid EEG findings may lead to better understanding of the development of a brain-computer music interface infrastructure capable of guiding the training, performance, and appreciation for emotional improvisatory status or actuating music interaction via emotional context.

Brain-Computer Music Interface, a bibliometric analysis

ABSTRACT This article depicts a bibliometric analysis done through visualization mechanisms and interpretation of bibliometric metadata on the research field of Brain-Computer Interface and Music or Brain-Computer Music Interface (BCMI). Citation, co-citation, co-authorship, and keywords co-occurrence analysis were carried out in this work in order to identify the intellectual structure, research trends, the organizations involved, and the methodological structure of such research field. The bibliometric metadata was visualized through VOSviewer and Scimat software. This study also includes the analysis of 227 papers done through 2005–2021 which include research and review articles, and proceedings papers. The results of this work demonstrate the growing and legitimizing of the research field, and the impact of the interdisciplinary work required in this area.

A Taxonomy of Musical Interfaces for Use in the Classroom

Instrument taxonomies have been a preoccupation throughout history. Musical interfaces which involve the use of a sound engine (which could be a software) to sonify and translate data from a user to produce a sound result, do not fit into traditional instrument taxonomies. Magnusson argues that it is necessary to determine classification principles of musical interfaces because there are clear benefits of such a taxonomy for inventors, performers, musicologists, and composers. We propose that educators are another group of people who can benefit from a taxonomy of musical interfaces which considers classifications relating to educational use. Several researchers have argued that there is a need to evaluate how and why technology is implemented in the music classroom. Previous taxonomies of musical interfaces have focused on design principals and user interaction and have only partially explored educational objectives. In this paper we propose a taxonomy of musical interfaces which builds on previous taxonomies but extends them into the educational field. Throughout four case studies of recent musical interfaces implemented in educational contexts in four continents, we show how this taxonomy can be employed to classify how and why a musical interface is implemented in the classroom to analyze the educational benefits they can promote.

Light.void~: An Autographic Case Study on Approaches, Constraints, and Affordances with a Light-dependent Musical Interface.

This paper discusses the strategies, considerations, and implications of designing and performing with a light-dependent digital musical interface (DMI), named light.void~. This interface is introduced as a replica of light thing, an existing DMI designed and popularized by British artist Leafcutter John. The rationale for reproducing this DMI is presented, followed by a discussion around the guiding criteria for establishing data-to-sound mappings, and the kind of affordances that these decisions may bring — including performer control, unpredictability, intentionality, spontaneity, action-sound reactivity, visual interest, and so on. The remainder of the paper focuses on dissecting the nature of this digital musical instrument, using contributions by DMI researchers Miranda and Wanderley as the main analytical framework. The outcome of this process is a semi-improvisational work titled «Umbra», along with the open source documentation for the light.void~ interface. Additionally, some relevant questions emerge with regards to performer expertise, observed vs. unobserved performance, as well as ontological frictions between instrument, composer, performer, designer, and audience.

Affective Brain-Computer Music Interface in Emotion Regulation and Neurofeedback: A Research Protocol

Introduction: Emotion regulation is an integral part of mental health, dynamically impacting brain function as one’s emotions change continuously throughout the day. Impairments in emotion regulation are associated with a range of psychiatric disorders. Although the implications of emotion regulation are crucial to mental health, few studies have examined training emotion regulation strategies with respect to the brain. Thus, this manuscript will propose an affective brain-computer music interface (aBCMI) prototype for emotion regulation that continuously generates music by estimating emotions from real-time electroencephalography (EEG) signals. Methods: In this proposal, we describe our prototype consisting of an emotion classifier that detects the expression of emotions from EEG signals, and a music generator that generates music reflective of those emotions. We evaluate the prototype in three separate studies. In study 1, the accuracy of the music generator is tested. In study 2, the accuracy of the emotion classifier is tested by assessing its correlation with real-time, self-reported emotions. In study 3, the generative music algorithm is assessed to explore emotion regulation strategies. Discussion: The proposed BCMI is expected to accurately estimate emotions, provide musical feedback of participants’ emotions, and enable users to intentionally modulate their emotions from musical feedback. This involves capturing the listener’s emotions in real-time using EEG signals, providing the opportunity to regulate one’s emotional state with musical feedback. Thus, in addition to enabling greater neurofeedback training of emotions, our prototype can enhance the understanding of affective computing and emotions with EEG and machine learning. Conclusion: Clinical applications of this prototype may have a tremendous impact as a neurofeedback tool in music therapy for training emotion regulation. Future research may benefit from using the proposed BCMI as a neurofeedback treatment in mood disorders.

POETRY/MUSIC INTERFACE IN “LONDON:” TOWARDS A MULTIMODAL SEMIOTIC READING

This paper entitled “Poetry/ Music Interface in “London:” Towards a Multimodal Semiotic Reading” aims at examining the way William Blake’s London was “re- written” by being both adopted and adapted for an animated song in 2008. It seeks to show first, that the traditional readings of Blake’s Works which analysed his poetry and visual art separately have failed to articulate their ontological dimensions and did not come to terms with the energetics of Blake’s oeuvre that consists mainly of creating parallel universes. The paper, later, focuses on the interface between the words, the image and the rhythm in Alex Robinson’s “London” (2008) which suggested a new way of understanding Blake’s London and the world in general. The choice of the mode through which to re-introduce what was written is at the crux of the “re-writing” process and music is one of the most persuasive modes in what Gunther Kress and Theo Van Leeuwen refer to as “multimodal communication.” The multimodal discourse accentuates the importance of the different modes of communication including the word, sound, image, rhythm and colour, and stresses the importance of their added meanings; hence my choice of multimodal semiotic analysis for this paper.

A General Framework for Visualization of Sound Collections in Musical Interfaces

While audio data play an increasingly central role in computer-based music production, interaction with large sound collections in most available music creation and production environments is very often still limited to scrolling long lists of file names. This paper describes a general framework for devising interactive applications based on the content-based visualization of sound collections. The proposed framework allows for a modular combination of different techniques for sound segmentation, analysis, and dimensionality reduction, using the reduced feature space for interactive applications. We analyze several prototypes presented in the literature and describe their limitations. We propose a more general framework that can be used flexibly to devise music creation interfaces. The proposed approach includes several novel contributions with respect to previously used pipelines, such as using unsupervised feature learning, content-based sound icons, and control of the output space layout. We present an implementation of the framework using the SuperCollider computer music language, and three example prototypes demonstrating its use for data-driven music interfaces. Our results demonstrate the potential of unsupervised machine learning and visualization for creative applications in computer music.

Affective computing in the context of music therapy: a systematic review

Music therapy is an effective tool to slow down the progress of dementia since interaction with music may evoke emotions that stimulates brain areas responsible for memory. This therapy is most successful when therapists provide adequate and personalized stimuli for each patient. This personalization is often hard. Thus, Artificial Intelligence (AI) methods may help in this task. This paper brings a systematic review of the literature in the field of affective computing in the context of music therapy. We particularly aim to assess AI methods to perform automatic emotion recognition applied to Human-Machine Musical Interfaces (HMMI). To perform the review, we conducted an automatic search in five of the main scientific databases on the fields of intelligent computing, engineering, and medicine. We search all papers released from 2016 and 2020, whose metadata, title or abstract contains the terms defined in the search string. The systematic review protocol resulted in the inclusion of 144 works from the 290 publications returned from the search. Through this review of the state-of-the-art, it was possible to list the current challenges in the automatic recognition of emotions. It was also possible to realize the potential of automatic emotion recognition to build non-invasive assistive solutions based on human-machine musical interfaces, as well as the artificial intelligence techniques in use in emotion recognition from multimodality data. Thus, machine learning for recognition of emotions from different data sources can be an important approach to optimize the clinical goals to be achieved through music therapy.

Research on the Emotions Based on Brain-Computer Technology: A Bibliometric Analysis and Research Agenda.

This study conducts a scientific analysis of 249 literature on the application of brain-computer technology in emotion research. We find that existing researches mainly focus on engineering, computer science, neurosciences neurology and psychology. PR China, United States, and Germany have the largest number of publications. Authors can be divided into four groups: real-time functional magnetic resonance imaging (rtfMRI) research group, brain-computer interface (BCI) impact factors analysis group, brain-computer music interfacing (BCMI) group, and user status research group. Clustering results can be divided into five categories, including external stimulus and event-related potential (ERP), electroencephalography (EEG), and information collection, support vector machine (SVM) and information processing, deep learning and emotion recognition, neurofeedback, and self-regulation. Based on prior researches, this study points out that individual differences, privacy risk, the extended study of BCI application scenarios and others deserve further research.

Breaking down the musician’s minds: How small changes in the musical instrument can impair your musical performance

The relationship between musicians and their musical instruments has influenced music engagement and musical structure across societies. In this work, we study how musicians react to changes in their instrument and the associations between keys and pitches using experiments that simulate the interface of the accordion. Seventeen accordionists, pianists and guitarists took part in the study. The results show accordion players are more affected by the changes in the musical interface than non-according players, for the same tasks. These observations support the extended cognition hypothesis, which proposes that coupled processes, such as the musician-instrument chain, count as an entire cognitive process.

Affective Brain-Computer Music Interfaces-Drivers and Implications.

Affective Brain-Computer Music Interfaces-Drivers and Implications.

Bidirectional Interactions With Humpback Whale Singer Using Concrete Sound Elements.

We describe an art–science project called “Feral Interactions—The Answer of the Humpback Whale” inspired by humpback whale songs and interactions between individuals based on mutual influences, learning process, or ranking in the dominance hierarchy. The aim was to build new sounds that can be used to initiate acoustic interactions with these whales, not in a one-way direction, as playbacks do, but in real interspecies exchanges. Thus, we investigated how the humpback whales generate sounds in order to better understand their abilities and limits. By carefully listening to their emitted vocalizations, we also describe their acoustic features and temporal structure, in a scientific way and also with a musical approach as it is done with musique concrète, in order to specify the types and the morphologies of whale sounds. The idea is to highlight the most precise information to generate our own sounds that will be suggested to the whales. Based on the approach developed in musique concrète, similarities with the sounds produced by bassoon were identified and then were processed to become “concrete sound elements.” This analysis also brought us to design a new music interface that allows us to create adapted musical phrases in real-time. With this approach, interactions will be possible in both directions, from and to whales.

An FCNN-Based Super-Resolution Mmwave Radar Framework for Contactless Musical Instrument Interface

In this article, we propose a framework for contactless human-computer interaction (HCI) using novel tracking techniques based on deep learning-based super-resolution and tracking algorithms. Our system offers unprecedented high-resolution tracking of hand position and motion characteristics by leveraging spatial and temporal features embedded in the reflected radar waveform. Rather than classifying samples from a predefined set of hand gestures, as common in existing work on deep learning with mmWave radar, our proposed imager employs a regressive full convolutional neural network (FCNN) approach to improve localization accuracy by spatial super-resolution. While the proposed techniques are suitable for a host of tracking applications, this article focuses on their application as a musical interface to demonstrate the robustness of the gesture sensing pipeline and deep learning signal processing chain. The user can control the instrument by varying the position and velocity of their hand above the vertically-facing sensor. By employing a commercially available multiple-input-multiple-output (MIMO) radar rather than a traditional optical sensor, our framework demonstrates the efficacy of the mmWave sensing modality for fine motion tracking and offers an elegant solution to a host of HCI tasks. Additionally, we provide a freely available software package and user interface for controlling the device, streaming the data to MATLAB in real-time, and increasing accessibility to the signal processing and device interface functionality utilized in this article.

Editors' Notes

Editors' Notes

More than ‘clips of dead fish’: gender, technologisation, and localisation in the amateur music videos of Wellington spearfishers

ABSTRACT This article explores the use of music in service of anthropocentric discourses and practices. We analyse spearfishing music (music synchronised with amateur online videos of spearfishing) as it has emerged in Wellington, Aotearoa/New Zealand. Through textual analysis of the gendered codes present in 40 locally produced videos, we interrogate how such practices exacerbate inequalities already present in music conventions, thus conjoining localised engagement with environmental issues with a broader discourse concerning gender representation and inequality. We further ground our position by providing qualitative perspectives attained through interviews and participant-observation ethnography. While we propose that the Anthropocene be understood as a predominantly masculinist and techno-normative framework, we also argue that the paradigms evident through spearfishing music interface with legacies of colonialism, capitalism, and Indigenous knowledge within the context of Aotearoa/New Zealand, which complicates the prospect of situating such texts and practices within a universalising account of the Anthropocene.