Digital Musical Instruments Research Articles

Shifting Ambiguity, Collapsing Indeterminacy: Designing with Data as Baradian Apparatus

This paper examines how digital systems designers distil the messiness and ambiguity of the world into concrete data that can be processed by computing systems. Using Karen Barad's agential realism as a guide, we explore how data is fundamentally entangled with the tools and theories of its measurement. We examine data-enabled artefacts acting as Baradian apparatuses: they do not exist independently of the phenomenon they seek to measure, but rather collect and co-produce observations from within their entangled state: the phenomenon and the apparatus co-constitute one another. Connecting Barad's quantum view of indeterminacy to the prevailing HCI discourse on the opportunities and challenges of ambiguity, we suggest that the very act of trying to stabilise a conceptual interpretation of data within an artefact has the paradoxical effect of amplifying and shifting ambiguity in interaction. We illustrate these ideas through three case studies from our own practices of designing digital musical instruments (DMIs). DMIs necessarily encode symbolic and music-theoretical knowledge as part of their internal operation, even as conceptual knowledge is not their intended outcome. In each case, we explore the nature of the apparatus, what phenomena it co-produces, and where the ambiguity lies to suggest approaches for design using these abstract theoretical frameworks.

A Fuzzy Gestural AI Approach for Expressive Interactivity in Multitouch Digital Musical Instruments Based on Laban Descriptors

Abstract One of the main research areas in the field of musical human–AI interactivity is how to incorporate expressiveness into interactive digital musical instruments (DMIs). In this study we analyzed gestures rooted in expressiveness by using AI techniques that can enhance the mapping stage of multitouch DMIs. This approach not only considers the geometric information of various gestures but also incorporates expressiveness, which is a crucial element of musicality. Our focus is specifically on multitouch DMIs, and we use expressive descriptors and a fuzzy logic model to mathematically analyze performers' finger movements. By incorporating commonly used features from the literature and adapting some of Rudolf Laban's descriptors—originally intended for full-body analysis—to finger-based multitouch systems, we aim to enrich the mapping process. To achieve this, we developed an AI algorithm based on a fuzzy control system that takes these descriptors as inputs and maps them to synthesis variables. This tool empowers DMI designers to define their own mapping rules based on expressive gestural descriptions, using musical metaphors in a simple and intuitive way. Through a user evaluation, we demonstrate the effectiveness of our approach in capturing and representing gestural expressiveness in the case of multitouch DMIs.

Exploring the Effects of Additional Vibration on the Perceived Quality of an Electric Cello

Haptic feedback holds the potential to enhance the engagement and expressivity of future digital and electric musical instruments. This study investigates the impact of artificial vibration on the perceived quality of a silent electric cello. We developed a haptic cello prototype capable of rendering vibration signals of varying degree of congruence with the produced sound. Experienced cellists participated in an experiment comparing setups with and without vibrotactile feedback, rating them on preference, perceived power, liveliness, and feel. Results show nuanced effects, with added vibrations moderately enhancing feel and liveliness, and significantly increasing perceived power when using vibrations obtained from the pickup at the cello’s bridge. High uncertainty in our statistical model parameters underscores substantial individual differences in the participants responses, as commonly found in qualitative assessments, and highlights the importance of consistent feedback in the vibrotactile and auditory channels. Our findings contribute valuable insights to the intersection of haptics and music technology, paving the way for creating richer and more engaging experiences with future musical instruments.

Analisis Pola Musik Karawitan di Tengah Era Digital

Traditional arts, especially musical music, are facing challenges in the digital era and the influence of foreign cultures. Efforts continue to be made to combine technology with traditional arts. The method adopted in this research is library research, which involves collecting data and information in depth through various references such as books, magazines, other references, and research results. This research explains the adaptation of Karawitan music patterns to modern technology and its impact on the interests of the younger generation. Use of digital musical instruments to create innovative compositions that combine traditional elements with contemporary elements. The role of the community in supporting the preservation of Karawitan art is also emphasized, with active participation in traditional music events and efforts to promote the culture to others. In conclusion, this article emphasizes the complexity of the relationship between tradition and innovation in the context of Karawitan music in the era of modern technology.

Eye-tracking digital music creation and performance: disability and ableism

ABSTRACT This paper focuses on the developmental process of eye trackers as accessible digital musical instruments (ADMIs) by outlining collaborative research that develops digital art and music creation and performance tools. These tools require eye movements only and are of interest to individuals with all types of mobility and particularly provide music-making options to users with limited mobility. Research grade and gaming eye-tracking technology is adapted with custom software to enable music creation and performance using eye movements only. The relationship of ableism to disability and the role of digital technology to counter the negative social forces of ableism are considered. Because eye-tracking art and music creation tools are rare outside research lab contexts, all users with the ability to move one eye – regardless of other physical ability – have pre-existing capability, which makes this work especially exciting in a disability context.

Audience’s Perceived Expectancy and Authenticity of Classical Music Performance by Digital Musical Instrument Mobile Apps

Classical music is characterized by its standardized practices and established patterns of audience behavior. Recent advancements in digital musical instrument (DMI) and mobile technology may potentially offer a means by which the art form can further evolve. This paper presents a study that aims to examine the audience’s perception of classical music concerts performed by ensembles using DMI mobile apps and tablets. A mixed-methods approach was employed, including a questionnaire survey administered to audiences (N = 141) attending a classical music concert featuring repertoire performed by DMI mobile apps ensembles, as well as semi-structured interviews conducted with a subset of participants (n = 13). The survey results revealed significant, strong, and positive correlations between participants’ enjoyment of listening to classical music, frequency of attending classical music concerts, expectancy, and authenticity. The interviews addressed relevant issues, such as differences in ensemble and instrumental practices, timbral quality, and the capabilities of DMI mobile apps. While there were negative perceptions regarding expressive body movements, emotional expression, and interaction among performers, positive feedback regarding the use of DMI mobile apps affirmed the potential to introduce innovative pedagogical approaches in school music education, cultivate a younger audience base, and bring fresh ideas to this genre.

The Theatre of “Glass Puppetry” Enhanced With The Use of The Technology of Lighting

The life of a Wayang Artist is an honorable role to have in the society of the Balinese people. Storytellers using the art of the wayang are highly respected, whose role in the community is to pass on the precious stories about the history, the culture and the spiritual heart of the Balinese people. The Wayang artist, known as the dalang goes on a life journey in sharing their philosophies through stories for the enrichment and soul of society. The process of internalization of the Puppet master from an early age through the world of wayang, forms a cognitive map of the creator who is immersed in the stories and characters of wayang art figures. The wayang artist continues to strive for innovation in form, exploring new techniques and exciting presentations, and with that in mind, the invention of the Glass Wayang enables the creator to explore the new technologies in developing a new way forward in the shadow puppet world. The process of creating Glass Puppets using the technology of lighting as a medium, opens one up to reflect on the mirroring of the self-image- to self-awareness and the inner workings our role in this human life Dramatic depictions of life processes are conveyed through the medium of glass puppets. Experiences and realities of life from the past until now. The search for identity is awakened. Wayang Kaca's work ranges from the making of various styles and forms of puppets supported by one hundred and six dancers accompanied by digital musical instruments (midi) and Bebonangan and Semarandhana gamelan. Digital musical instruments (midi) and gamelan are not only accompaniment but are an important part in providing a traditional and modern feel. The performance venue at Sanggar Paripurna Bona, Gianyar includes a side stage along with the traditional proscenium stage. allowing for a full presentation for displaying this Glass Wayang work. The facilities the sanggar offers allow for all aspects of the creation process in developing, rehearsing, and presenting a final performance.

Analysis of Accessible Digital Musical Instruments through the lens of disability models: a case study with instruments targeting d/Deaf people

Music educators and researchers have grown increasingly aware of the need for traditional musical practices to promote inclusive music for disabled people. Inclusive music participation has been addressed by Accessible Digital Musical Instruments (ADMIs), which welcome different ways of playing and perceiving music, with considerable impact on music-making for disabled people. ADMIs offer exciting possibilities for instrument design to consider and incorporate individual constraints (e.g., missing arm, low vision, hearing loss, etc.) more than traditional acoustic instruments, whose generally fixed design allows little room for disabled musicians inclusivity. Relatively few works discuss ADMIs in the context of disability studies, and no work has investigated the impact of different disability models in the process of designing inclusive music technology. This paper proposes criteria to classify ADMIs according to the medical, social, and cultural models of disability, then applies these criteria to evaluate eleven ADMIs targeting d/Deaf people. This analysis allows us to reflect on the design of ADMIs from different perspectives of disability, giving insights for future projects and deepening our understanding of medical, social, and cultural aspects of accessible music technology.

Music for All: An Intervention Project in an Artistic School in Portugal

In Portugal, in 2018, an action-research project began to find solutions so that children with cerebral palsy (CP) could learn music in Arts Education Programmes of Music (AEPM). When conducting the characterisation of the child with CP, which gave rise to the study, and a series of literature reviews to find out what has been done in this area, we came across several works related to different Accessible Digital Musical Instruments (ADMIs) which can be employed by users with different needs. In the present case of the child under study, two of these ADMIs were chosen and used because of the opportunities they offer: Netytar and Netychords. These instruments allowed the child to access an artistic school near their residence after the respective admission test. In the Intervention Project (IP), which lasted six months, we analysed how the entry process was carried out, so we qualitatively studied the music teachers' field diary records. This situation included several interviews and conversations with the educational community involved. More specifically, the following were analysed: (i) the process of the child's entry into the artistic school; (ii) the work done by the piano teacher in the first three months of the study; (iii) the curricular adaptations proposed by the music teachers (MTs) of the ensemble and music training class so that the child could fulfil the proposed objectives; and (iv) the arrangements and musical compositions adapted so that the child could fulfil the demands of the school programme. As a result, we noticed that the peers of the child with CP adapted much better to the participation processes in favour of inclusion than the adults involved, probably because among the adults there was fear in the face of a still unknown situation, as is the case of working with a child who accesses information and music teaching unusually. We also noticed the urgent need for specific training and supervision among all involved - MTs, operational assistants (OAs), and parents/guardians (PGs). We were faced with the need for training in minor details such as accompanying the child with CP to move and use the toilet. In other words, it is necessary to prevent and prepare, empowering those involved with a series of skills so that children with special needs can fully and actively participate in this type of education.

The development of a Modular Accessible Musical Instrument Technology Toolkit using action research

Within the field of digital musical instruments, there have been a growing number of technological developments aimed at addressing the issue of accessibility to music-making for disabled people. This study summarizes the development of one such technological system—The Modular Accessible Musical Instrument Technology Toolkit (MAMI Tech Toolkit). The four tools in the toolkit and accompanying software were developed over 5 years using an action research methodology. A range of stakeholders across four research sites were involved in the development. This study outlines the methodological process, the stakeholder involvement, and how the data were used to inform the design of the toolkit. The accessibility of the toolkit is also discussed alongside findings that have emerged from the process. This study adds to the established canon of research around accessible digital musical instruments by documenting the creation of an accessible toolkit grounded in both theory and practical application of third-wave human–computer interaction methods. This study contributes to the discourse around the use of participatory and iterative methods to explore issues with, and barriers to, active music-making with music technology. Outlined is the development of each of the novel tools in the toolkit, the functionality they offer, as well as the accessibility issues they address. The study advances knowledge around active music-making using music technology, as well as in working with diverse users to create these new types of systems.

What we can do with sounds

Auditory perception can be dynamic when sound sources are in motion or changing, when the listener is moving relative to the source of sound, or both. Moreover, our engagement with the acoustic environment is often active rather than passive. That is, sounds can be brought into use to support and guide perceptual-motor behaviour. This presentation will review examples of recent empirical work on the role of auditory perception in the control of motor behaviour: what we can do with sounds. Data from experiments on intercepting moving objects without vision, acoustic navigation and orientation in people with visual impairments, exploration and learning of novel digital musical instruments, and rhythmic sounds in gait rehabilitation, all demonstrate different ways that sounds can convey action-relevant information for skilful listeners. Furthermore, they point to a need to consider the auditory system not merely as a passive antenna for inert distal stimuli, but as integrated within an active meaning-seeking organism interacting with a dynamic environment.

Digital Musical Instruments in Special Educational Needs Schools: Requirements from the Music Teachers’ Perspective and the Status Quo in Germany

Digital musical instruments (DMIs) offer the possibility to create barrier-free access to active music-making and to unique sound aesthetics for a broad group of people, including those who experience disabling barriers to access when using traditional acoustic musical instruments. However, current music education research focuses primarily on app-based DMIs. Although these devices have numerous advantages, such as ubiquitous availability and flexibility, they may be experienced as disabling by people with severe cognitive or complex disabilities. Thus, they only partially exploit the potential of DMIs for special educational needs (SEN) music practices, which we outline in this article. However, given that no comprehensive studies have yet been conducted on the use of DMIs in SEN schools, the actual motives and barriers for SEN school teachers to employ different types of DMIs in class are largely unknown. To address this research gap, we present the results of a quantitative survey covering all SEN schools in 12 of Germany's 16 federal states. We surveyed the status quo of DMI use in SEN schools and the perceived potential of DMIs and DMI-related information needs from the music teachers’ perspective. Our findings demonstrate that DMIs are only rarely used in Germany, with the exception of established standard DMIs such as keyboards and music apps. Unfortunately, accessible DMIs (ADMIs) are hardly used. Related to the rare use of DMIs in SEN schools, we also identified a lack of domain-specific knowledge among music teachers and concluded that there was a need to develop DMIs specifically designed for use in classroom education. Finally, we discuss the potential of using open-source DMI technology as well as the importance of identifying music teachers’ attitudes during DMI development.

A deep learning-based mathematical modeling strategy for classifying musical genres in musical industry

Abstract Since the beginning of the digital music era, the number of available digital music resources has skyrocketed. The genre of music is a significant classification to use when elaborating music; the role of music tags in locating and categorizing electronic music services is essential. To categorize such a large music archive manually would be prohibitively expensive and time-consuming, rendering it obsolete. This study’s main contributions to knowledge are the following: This article will break down the music into many MIDI (music played on a digital musical instrument) movements, playing way close by analysis movement, character extraction from passages, and character sequencing from movement so that you may get a clearer picture of what you are hearing. The procedure includes the following steps: extracting the note character matrix, extracting the subject and segmentation grouping based on the note character matrix, researching and extracting beneficial characteristics based on the theme of the segments, and composing the feature sequence. It is challenging for the sorter to acquire spatial and contextual knowledge about music using traditional classification techniques due to its shallow structure. This study uses the unique pattern of input MIDI segments, which are used to probe the relationship between recurrent neural networks and attention. The approach for music classification is verified when paired with the testing precision of the same-length segment categorization; thus, gathering MIDI tracks 1920 along with genre tags from the network to construct statistics sets and perform music classification analysis.

Collaborative music-making: special educational needs school assistants as facilitators in performances with accessible digital musical instruments

The field of research dedicated to Accessible Digital Musical Instruments (ADMIs) is growing and there is an increased interest in promoting diversity and inclusion in music-making. We have designed a novel system built into previously tested ADMIs that aims at involving assistants, students with Profound and Multiple Learning Disabilities (PMLD), and a professional musician in playing music together. In this study the system is evaluated in a workshop setting using quantitative as well as qualitative methods. One of the main findings was that the sounds from the ADMIs added to the musical context without making errors that impacted the music negatively even when the assistants mentioned experiencing a split between attending to different tasks, and a feeling of insecurity toward their musical contribution. We discuss the results in terms of how we perceive them as drivers or barriers toward reaching our overarching goal of organizing a joint concert that brings together students from the SEN school with students from a music school with a specific focus on traditional orchestral instruments. Our study highlights how a system of networked and synchronized ADMIs could be conceptualized to include assistants more actively in collaborative music-making, as well as design considerations that support them as facilitators.

Design and Assessment of Digital Musical Devices Yielding Vibrotactile Feedback

Touch has a pivotal importance in determining the expressivity of musical performance for a number of musical instruments. However, most digital musical devices provide no interactive force and/or vibratory feedback to the performer, thus depriving the somatosensory channel of a number of cues. Is the lack of haptic feedback only an aesthetic issue, or does it remove cues essential for digital instrument playing? If so, at which level is the interaction objectively impoverished? What are the effects on musical performance? In this survey article we illustrate our recent research about the use of vibrotactile feedback in three digital instrument interfaces, using tools that we developed over several years and made available to the community in open-source form. These interfaces span a wide range of familiarity and gestural opportunities, enabling us to explore the impact of haptic feedback on different types of digital instruments. We conducted experiments with professional musicians to assess the impact of vibratory cues on both the perceived quality of the instrument and the playing experience, as well as on musical performance. Particular attention was paid to scientific rigor and repeatability of the results, so as to serve as a reference for researchers and practitioners of the musical haptics community. Our results suggest a significant role of vibrotactile feedback in shaping the perception of digital musical instruments, although the effects on musical performance varied depending on the interfaces tested.

On the Organological Potential of Video Games

The various—and often unpredictable—musical applications given to new digital technologies have fundamentally altered our ways of creating, performing, representing, and understanding musical instruments, giving way to new sounding artifacts that pose unprecedented challenges to musical iconography and organology. The multifaceted nature of digital musical instruments has exposed the limitations of traditional organology, prompting a redefinition of the concept of instrumentality and an upsurge of new methodologies. This renovation, however, has yet to recognize the organological potential of video games, which often make use of visual depictions and reconfigurations of musical instruments in the creation and characterization of virtual universes and characters. Musical instruments play an essential role in regulating the tensions between fiction and reality that underlie videoludic experiences by constituting an active agent in the player’s interaction and emerging relationships with the gameworld and its music. Research on these artifacts has been scarce yet promising thus far, providing this article with a wide-ranging collection of references that have served to inform the proposal of a new cataloguing card model for the identification, examination, and classification of musical instruments represented in, and constituted by, video games and consoles. This model merges traditional organological knowledge inherited from hierarchical systems of classification—such as the Hornbostel-Sachs—with new paradigms and taxonomical categories. In so doing, this model aims to accommodate the dual nature of ludomusical instruments as both iconographical resources of undeniable sociocultural transcendence, and organological artifacts with a set of assigned roles within their virtual worlds. The ultimate goal is to provide a framework that fully recognizes the impact of ludomusical instruments in musical experiences and organological knowledge and expectations both inside and outside the medium.

HapticSOUND: An Interactive Learning Experience with a Digital Musical Instrument

In this paper, an interactive learning experience is proposed, aiming to involve museum visitors in a personalized experience of the transmittal of cultural knowledge in an active and creative way. The proposed system, called HapticSOUND, consists of three subsystems: (a) the Information, where visitors are informed about the traditional musical instruments; (b) the Entertainment, where visitors are entertained by playing serious games to virtually assemble traditional musical instruments by a set of 3D objects; and (c) the Interaction, where visitors interact with a digital musical instrument which is an exact 3D-printed replica of a traditional musical instrument, where cameras have been placed to capture user gestures and machine learning algorithms have been implemented for gesture recognition. The museum visitor can interact with the lifelike replica to tactilely and aurally explore the instrument’s abilities, producing sounds guided by the system and receiving real-time visual and audio feedback. Emphasis is given to the Interaction Subsystem, where a pilot study was conducted to evaluate the usability of the subsystem. Preliminary results were promising since the usability was satisfactory, indicating that it is an innovative approach that utilizes sensorimotor learning and machine learning techniques in the context of playing sounds based on real-time gesture and fingering recognition.

An End-to-End Musical Instrument System That Translates Electromyogram Biosignals to Synthesized Sound

Abstract This article presents a custom system combining hardware and software that senses physiological signals of the performer's body resulting from muscle contraction and translates them to computer-synthesized sound. Our goal was to build upon the history of research in the field to develop a complete, integrated system that could be used by nonspecialist musicians. We describe the Embodied AudioVisual Interaction Electromyogram, an end-to-end system spanning wearable sensing on the musician's body, custom microcontroller-based biosignal acquisition hardware, machine learning–based gesture-to-sound mapping middleware, and software-based granular synthesis sound output. A novel hardware design digitizes the electromyogram signals from the muscle with minimal analog preprocessing and treats it in an audio signal-processing chain as a class-compliant audio and wireless MIDI interface. The mapping layer implements an interactive machine learning workflow in a reinforcement learning configuration and can map gesture features to auditory metadata in a multidimensional information space. The system adapts existing machine learning and synthesis modules to work with the hardware, resulting in an integrated, end-to-end system. We explore its potential as a digital musical instrument through a series of public presentations and concert performances by a range of musical practitioners.

Chaos Bells: Instrument Size and Entangled Music Performance

ABSTRACT There is a current trend in the instrument design industry of scaling down physical dimensions, resulting in smaller and smaller ‘desktop’ instruments, but research conducted with large interfaces reveals the role of instrument size on music performance. Chaos Bells is a very large digital musical instrument designed with both artistic and analytical goals in mind: it is a probe into the effect of instrument size on performance, while also being a vehicle for the first author’s performance practice. Drawing on entanglement theories of HCI we supplement the findings of lab-based studies with Chaos Bells with new discoveries made by the first author when touring with Chaos Bells. This research elucidates the indirect influences of stakeholders on instrument design culture, and ways that taking a large instrument design approach can benefit designers of instruments of all sizes.

Real-time musical performance across and within extended reality environments

The design of real-time digital musical instruments, both novel as well as the emulation of traditional musical performance practices using computer systems, has always required nearly imperceptible latencies between controlling gesture and expressed sound. Sound and music based experiences built within immersive visual rendered spaces often struggle to provide seamless articulate control of rendered dynamic output. Working across and within extended reality environments, the complexity of the challenge continues to evolve with current generations of hardware and software pushing core and graphics processors to their respective limits. The introduction of networked performance spaces has only added to the complexity, resulting in the need for additional latency mitigation strategies from both the technical and artistic domains. This paper will discuss recent creative and technical strategies and concerns in the design, development and use of real-time virtual musical instruments.