Digital Musical Instruments Research Articles

An ubiquitous smart guitar system for collaborative musical practice

ABSTRACTUbiquitous music (UbiMus) proposes to study how social interaction with mobile and distributed technologies can converge to form novel creativity support tools and music artistic practices. A recent field overlapping with UbiMus is the Internet of Musical Things, which refers to ecosystems of interconnected embedded computers (Musical Things) enabling users to produce, interact with or experience musical content. Musical Things embed electronics, sensors, data forwarding and processing software into physical or virtual objects. Smart musical instruments (SMIs) are an emerging class of Musical Things provided with capabilities of capturing and receiving data supporting instrumental musical practice. Due to their portability and self-containdeness, SMIs enable novel ubiquitous interactions between performers of acoustic and digital musical instruments. After a review of current trends in SMI research, we propose an ubiquitous smart guitar system which uses the guitar as a hub for collaborative music making. We then present a survey conducted with 18 performers to assess the usability, creativity support and engagement with the system. Results show a positive emotional engagement with the system which overall was found easy to use and novel. We also discuss several barriers to creative interaction related to the size of the user interface, creative agency and personalisation.

Preparing the Digital Piano: Introducing bitKlavier

Abstract This article describes a new kind of digital musical instrument, a novel assemblage of the familiar MIDI keyboard with custom interactive software. Inspired by John Cage's prepared piano, our instrument both takes advantage of and subverts the pianist's hard-earned, embodied training, while also inviting an extended configuration stage that “prepares” the instrument to behave in composition-specific, idiosyncratic ways. Through its flexible—though constrained—design, the instrument aims to inspire a playful approach to instrument building, composition, and performance. We outline the development history of our instrument, called bitKlavier, its current design, and some of its musical possibilities.

Nonlinear dynamical processes in musical interactions: Investigating the role of nonlinear dynamics in supporting surprise and exploration in interactions with digital musical instruments

Nonlinear dynamical processes play a central role in many acoustic instruments, yet they rarely feature in digital instruments, and are little understood from an interaction design perspective. Such processes exhibit behaviours that are complex, time-dependent, and chaotic, yet in the context of acoustic instruments can facilitate interactions that are explorable, learnable and repeatable. This suggests that these processes merit deeper investigation for digital music interaction design. Two studies are presented which investigate user interaction with nonlinear dynamical musical tools. A lab-based study used four purpose-built digital musical instruments to test interaction designs featuring nonlinear dynamical processes. Evaluations with 28 musicians demonstrated the potential for these processes to provoke creative surprises, and support exploration without a corresponding loss of control. A subsequent ethnographically-informed study with 24 musicians linked these findings to a mode of engagement which we term ‘edge-like interaction’. Edge-like interactions draw on the complex, unpredictable behaviours found in nonlinear dynamical processes close to critical thresholds, facilitating creative exploration. The two complementary studies provide evidence both for the existing importance of nonlinear dynamical processes in musical interactions with acoustic interactions, and their potential for deployment in the development of new creative digital technologies, musical or otherwise.

Mapping by Observation: Building a User-Tailored Conducting System From Spontaneous Movements

Metaphors are commonly used in interface design within Human-Computer Interaction (HCI). Interface metaphors provide users with a way to interact with the computer that resembles a known activity, giving instantaneous knowledge or intuition about how the interaction works. A widely used one in Digital Musical Instruments (DMIs) is the conductor-orchestra metaphor, where the orchestra is considered as an instrument controlled by the movements of the conductor. We propose a DMI based on the conductor metaphor that allows to control tempo and dynamics and adapts its mapping specifically for each user by observing spontaneous conducting movements (i.e., movements performed on top of fixed music without any instructions). We refer to this as mapping by observation given that, even though the systemis trained specifically for each user, this training is not done explicitly and consciously by the user. More specifically, the system adapts its mapping based on the tendency of the user to anticipate or fall behind the beat and observing the Motion Capture descriptors that best correlate to loudness during spontaneous conducting. We evaluate the proposed system in an experiment with twenty four (24) participants where we compare it with a baseline that does not perform this user-specific adaptation. The comparison is done in a context where the user does not receive instructions and, instead, is allowed to discover by playing. We evaluate objective and subjective measures from tasks where participants have to make the orchestra play at different loudness levels or in synchrony with a metronome. Results of the experiment prove that the usability of the system that automatically learns its mapping from spontaneous movements is better both in terms of providing a more intuitive control over loudness and a more precise control over beat timing. Interestingly, the results also show a strong correlation betweenmeasures taken fromthe data used for training and the improvement introduced by the adapting system. This indicates that it is possible to estimate in advance how useful the observation of spontaneous movements is to build user-specific adaptations. This opens interesting directions for creating more intuitive and expressive DMIs, particularly in public installations.

The construction and evaluation of a musical glove

A digital musical instrument is different from an acoustic one because its gesture controllers are decoupled from the sound synthesis. Because of this, it is possible to separately design the control interface and the sound synthesis independently, and then digitally implement the gesture-sound mapping. This allows diverse possibilities for musical expression. A particular kind of digital musical instruments are musical gloves. They can capture the hand gestures, which are later mapped into sounds. By means of electronic sensors and digital sound synthesis, this work consisted of building a musical glove. In the development of the work the gesture-sound mapping and sound possibilities were explored in an embedded system with low computational resources.

Smart Musical Instruments: Vision, Design Principles, and Future Directions

Smart musical instruments (SMIs) are a family of Internet of Musical Things devices for music creation. They are characterized by sensors, actuators, embedded intelligence, and wireless connectivity to local networks and to the Internet. In this paper, we depict a vision for this recent research area, which merges the research fields on digital musical instruments and smart objects, and fosters new types of interactions between the player and the instrument, between the player and other players, and between the player and audience members. We propose a set of capabilities and technical features characterizing SMIs, as well as a design for a technological architecture supporting them. To illustrate possible applications enabled by this vision, we present a set of scenarios exploiting the intelligence embedded in SMIs. We also propose a set of guidelines that can help digital luthiers in the process of designing SMIs. Finally, we present a set of directions for future research.

Third City 2017: Improvisational Roles in Performances Using Live Sampling

Abstract The 2017 set by the electroacoustic duo Third City comprised five pieces, each defined by an audio path linking different acoustic musical instruments to digital musical instruments to enable live sampling. Performances were then improvised within structures developed in rehearsal. The authors here ask how the different instruments and audio paths influenced the improvisational roles taken by the performers. Previously established differences between acoustic musical instruments and digital musical instruments are highlighted, and questions regarding their use within improvisation are articulated. A taxonomy of improvisational roles is then selected and applied to the pieces. In identifying correlations between the instruments and audio paths of the five pieces and the improvisational roles used by the performers, conclusions are reached to serve as guidance in the setting up of audio paths for other electroacoustic improvisation pieces using live sampling. This article is the result of research into practice, an asynchronous post hoc consideration (Onsman and Burke 210) of the 2017 Third City set carried out by the duo having repositioned themselves relative to their music-making selves as researchers referring to both the experience of performers and the projected experience of the audience as inferred from archive footage.

Exploring the motivations for building new digital musical instruments

Over the past four decades, the number, diversity and complexity of digital musical instruments (DMIs) has increased rapidly. There are very few constraints on DMI design as such systems can be easily reconfigured, offering near limitless flexibility for music-making. Given that new acoustic musical instruments have in many cases been created in response to the limitations of available technologies, what motivates the development of new DMIs? We conducted an interview study with ten designers of new DMIs, in order to explore (a) the motivations electronic musicians may have for wanting to build their own instruments; and (b) the extent to which these motivations relate to the context in which the artist works and performs (academic vs club settings). We found that four categories of motivation were mentioned most often: M1 – wanting to bring greater embodiment to the activity of performing and producing electronic music; M2 – wanting to improve audience experiences of DMI performances; M3 – wanting to develop new sounds, and M4 – wanting to build responsive systems for improvisation. There were also some detectable trends in motivation according to the context in which the artists work and perform. Our results offer the first systematically gathered insights into the motivations for new DMI design. It appears that the challenges of controlling digital sound synthesis drive the development of new DMIs, rather than the shortcomings of any one particular design or existing technology.

Hackable Instruments: Supporting Appropriation and Modification in Digital Musical Interaction

This paper investigates the appropriation of digital musical instruments, wherein the performer develops a personal working relationship with an instrument that may differ from the designer's intent. Two studies are presented which explore different facets of appropriation. First, a highly restrictive instrument was designed to assess the effects of constraint on unexpected creative use. Second, a digital instrument was created which initially shared several constraints and interaction modalities with the first instrument, but which could be rewired by the performer to discover sounds not directly anticipated by the designers. Each instrument was studied with 10 musicians working individually to prepare public performances on the instrument. The results suggest that constrained musical interactions can promote the discovery of unusual and idiosyncratic playing techniques, and that tighter constraints may paradoxically lead to a richer performer experience. The diversity of ways in which the rewirable instrument was modified and used indicates that its design is open to interpretation by the performer, who may discover interaction modalities that were not anticipated by the designers.

Digital instruments and their uses in music therapy

ABSTRACTTechnology could represent an ally in easing the formulation of a bond between music therapist and client. So far, both scholars and therapists have claimed that the implementation of technology could be paramount in transforming the client’s therapeutic process, beyond representing a new and effective methodology for session analysis. This article proposes the implementation of musical technologies within the daily practice of music therapy, the latter understood as the appropriate field for the application of interactive systems technology, which track the user’s movements and transform them into audiovisual and haptic feedback. The article presents an exploratory review of the use of digital instruments, based on motion tracking, in order to develop a new basis for music therapy procedures, and discusses how specific features of digital musical instruments could benefit music therapy sessions. All discussions are set within the framework of embodied cognition. We provide a discussion on how multisensory integration may be used in the treatment of clients with severe cognitive and motor difficulties. We advocate the need for an interdisciplinary approach to the practice of music therapy, and propose future lines of research concerned with the design of multimodal and empowerment-based technologies.

Action-sound Latency and the Perceived Quality of Digital Musical Instruments

Asynchrony between tactile and auditory feedback (action-sound latency) when playing a musical instrument is widely recognized as disruptive to musical performance. In this paper we present a study that assesses the effects of delayed auditory feedback on the timing accuracy and judgments of instrument quality for two groups of participants: professional percussionists and non-percussionist amateur musicians. The amounts of delay tested in this study are relatively small in comparison to similar studies of auditory delays in a musical context (0 ms, 10 ms, 10 ms ± 3 ms, 20 ms). We found that both groups rated the zero latency condition as higher quality for a series of quality measures in comparison to 10 ms ± 3 ms and 20 ms latency, but did not show a significant difference in rating between 10 ms latency and zero latency. Professional percussionists were more aware of the latency conditions and showed less variation of timing under the latency conditions, although this ability decreased as the temporal demands of the task increased. We compare our findings from each group and discuss them in relation to latency in interactive digital systems more generally and experimentally similar work on sensorimotor control and rhythmic performance.

Extended Playing Techniques on an Augmented Virtual Percussion Instrument

Abstract Innovation and tradition are two fundamental factors in the design of new digital musical instruments. Although apparently mutually exclusive, novelty does not imply a total disconnection from what we have inherited from hundreds of years of traditional design, and the balance of these two factors often determines the overall quality of an instrument. Inspired by this rationale, in this article we introduce the Hyper Drumhead, a novel augmented virtual instrument whose design is deeply rooted in traditional musical paradigms, yet aimed at the exploration of unprecedented sounds and control. In the first part of the article we analyze the concepts of designing an augmented virtual instrument, explaining their connection with the practice of augmenting traditional instruments. Then we describe the design of the Hyper Drumhead in detail, focusing on its innovative physical modeling implementation. The finite-difference time-domain solver that we use runs on the parallel cores of a commercially available graphics card and permits the simulation of real-time 2-D wave propagation in massively sized domains. Thanks to the modularity of this implementation, musicians can create several 2-D virtual percussive instruments that support realistic playing techniques but whose affordances can be enhanced beyond most of the limits of traditional augmentation.

Facilitating Accessibility in Performance: Participatory Design for Digital Musical Instruments

Facilitating Accessibility in Performance: Participatory Design for Digital Musical Instruments

Vocal Control of Sound Synthesis Personalized by Unsupervised Machine Listening and Learning

AbstractIn this article we describe a user-driven adaptive method to control the sonic response of digital musical instruments using information extracted from the timbre of the human voice. The mapping between heterogeneous attributes of the input and output timbres is determined from data collected through machine-listening techniques and then processed by unsupervised machine-learning algorithms. This approach is based on a minimum-loss mapping that hides any synthesizer-specific parameters and that maps the vocal interaction directly to perceptual characteristics of the generated sound. The mapping adapts to the dynamics detected in the voice and maximizes the timbral space covered by the sound synthesizer. The strategies for mapping vocal control to perceptual timbral features and for automating the customization of vocal interfaces for different users and synthesizers, in general, are evaluated through a variety of qualitative and quantitative methods.

Gesture-sound causality from the audience’s perspective: Investigating the aesthetic experience of performances with digital musical instruments.

In contrast to their traditional, acoustic counterparts, digital musical instruments (DMIs) rarely feature a clear, causal relationship between the performer’s actions and the sounds produced. They often function simply as systems for controlling digital sound synthesis, triggering computer-generated audio. This study aims to shed light on how the level of perceived causality of DMI designs impacts audience members’ aesthetic responses to new DMIs. In a preliminary survey, 49 concert attendees listed adjectives that described their experience of a number of DMI performances. In a subsequent experiment, 31 participants rated video clips of performances with DMIs with causal and acausal mapping designs using the eight most popular adjectives from the preliminary survey. The experimental stimuli were presented in their original version and in a manipulated version with a reduced level of gesture-sound causality. The manipulated version was created by placing the audio track of one section of the recording over the video track of a different section. It was predicted that the causal DMIs would be rated more positively, with the manipulation having a stronger effect on the ratings for the causal DMIs. Our results confirmed these hypotheses, and indicate that a lack of perceptible causality does have a negative impact on ratings of DMI performances. The acausal group received no significant difference in ratings between original and manipulated clips. We posit that this result arises from the greater understanding that clearer gesture-sound causality offers spectators. The implications of this result for DMI design and practice are discussed. (PsycINFO Database Record (c) 2018 APA, all rights reserved)

Virtual kompang: mapping in-air hand gestures for music interaction using gestural musical controller

The introduction of new gesture interfaces has been expanding the possibilities of creating new Digital Musical Instruments (DMI). However, the created interfaces are mainly focused on modern western musical instruments such as piano, drum and guitar. This paper presents a virtual musical instrument, namely Virtual Kompang, a traditional Malay percussion instrument. The interface design and its implementation are presented in this paper. The results of a guessability study are presented in the study to elicit end-user hand movement to map onto commands. The study demonstrated the existing of common hand gestures among the users on mapping with the selected commands. A consensus set of gestures is presented as the outcome of this study.Keywords: Digital Music Instrument, Virtual Environment, Gestural Control, Leap Motion, Virtual Instrument

Uncanny Materialities: Digital Strategies for Staging Supernatural Themes Drawn from Medieval Ballads

In the medieval tradition of ballads, a recurring theme is that of transformation. In a staged concert for chamber orchestra, singers and dancers called Varelser och Ballader ( Beings and Ballads), we explored this theme using ballads coupled with contemporary poetry and new music. The performance made use of custom-made digital musical instruments, using video analysis and large-scale physical interfaces for transformative purposes. In this article, we describe the piece itself as well as how uncanny qualities of the digital were used to emphasize eerie themes of transformation and deception by the supernatural beings found in the medieval ballads.

Physical Intentions: Exploring Michel Waisvisz’s The Hands (Movement 1)

Through a musical examination of The Hands (Movement 1) (1986), this article reveals musical developments of Michel Waisvisz’s early performance practice with The Hands. Waisvisz’s digital musical instrument, The Hands, helped shape the body as a recognisable agent of musical structure; this article details how the instrument controls relate to musical gesture and reveals the physical efforts of the composer through spectral analyses of the sound recording and an investigation of a 1987 performance video. The rich context of The Hands – the instrument controls, Waisvisz’s decades-long dedication to the instrument’s potential and, more centrally, the exploration of human effort in performance – have made this pioneering live-performance system for electronic music stand for more than just a musical instrument. Many authors cite Michel Waisvisz’s Hands technology but few provide musical analyses of Waisvisz’s music. The musical analysis of The Hands (Movement 1) (1986) highlights the art practice of a seminal composer–performer through one of his most important digital musical instruments and signals how the body became an instrument for playing digital electronic music.

Applying information theory to investigate music performance with continuous electronic sensors

A wide variety of electronic sensors can be used for designing new digital musical instruments and other human-computer interfaces. However, presently human abilities for continuously controlling such sensors are not well quantified. The field of information theory suggests that a human together with a user interface can be modeled as a communication channel. Previously, Fitts' Law used a discrete communications channel to model information conveyed by a human pointing at discrete targets. In contrast, the present work employs a continuous communications channel to model a human continuously controlling an analog-valued sensor. The Shannon-Hartley theorem implies that the channel capacity (e.g., HCI throughput) can be estimated by asking human subjects to perform gestures that match idealized, bandlimited Gaussian “target gestures” across a range of bandwidths. Then, the signal-to-noise ratio of the recorded gestures determines the channel capacity (e.g., HCI throughput). This approach is tested on human users alternately operating simple analog sensors. Suggestions are made for creating knowledge about user interfaces that could potentially transmit an enhanced amount of information to a computer.

Bela: An embedded platform for low-latency feedback control of sound

Bela is an open-source embedded platform for audio and sensor processing. It uses a Beagle Bone Black single-board computer with a custom hard-real time audio environment based on Xenomai Linux, which is capable of submillisecond round-trip audio latencies (as low as 80 microseconds in certain configurations). The Bela hardware features stereo audio input and output, 8 channels each of 16-bit analog input and output, and 16 digital I/Os, all sampled at audio rates with nearly jitter-free alignment to the audio clock. This paper will present the hardware, software, and selected applications of Bela. Bela is suitable for creating digital musical instruments and interactive audio systems, and its low latency makes it especially well adapted for real-time feedback control over acoustic systems. It has been used in feedback control experiments with wind and string instruments and used as the basis for a study of the performer's experience of latency on percussion instruments.