Gesture Performance Research Articles

Hapstick-Figure: Investigating the Design of a Haptic Representation of Human Gestures from Theater Performances for Blind and Visually-Impaired People

Theaters serve as platforms that transport audiences into diverse worlds, offering a collective enjoyment of live performances and a shared cultural experience. However, theater performances have strong visual components, such as physical props and actors’ movements and gestures, which are inaccessible to visually impaired and blind audience members and thus can exclude them from such shared social experiences. We conducted formative interviews with eight blind and visually impaired people about their experiences with barriers to theater performance gestures. We then present Hapstick-Figure, a prototype design to represent and communicate human gestures via a 3D-printed tactile surface. Next, we used Hapstick-Figure as a technology probe in a qualitative evaluation with six of our BVI participants to explore non-visual interpretation and engagement with this prototype. We outline insights into the haptic representation of theater performance gestures and reflections on designing for accessibility in this context.

A study of gestures representing musical emotions

The creation of music has originated from the burst of inspiration of the composer. Many studies suggested that improvisation plays a more vital role than composition in the emergence of new forms of music. The production of music, especially in Western cultures, tends to follow its traditional music theory. With the emergence of Pythagorean music theory, Western musicology has been defined and developed in depth. The composer’s creation is based on music creation within the framework of the Western music theory. However, in recent years, with the rise of the topic of music gestures, some scholars have learned that music ontology as a four-dimensional hypercube is inseparable from the performer’s gesture. Therefore, some musicians also attempt to create flexible music from the perspective of music performers and based on performance gestures. This kind of music creation breaks the traditional music theory system and has more personal style characteristics, but few people can understand the thinking logic behind this kind of creative behavior. Therefore, this article focuses on the singing gesture and reversely analyzes its impact on musical creation. We use a recent mathematic definition of gestures and we discuss emotion from its etymological origin (from the Latin e-movere) as a gestural movement from inside out. This article conducts an in-depth analysis based on the singer’s gesture for the following two reasons: First, the literature on the connection between singing gesture theory and composition theory remains scarce. Second, gestures produced by a singer are the only performance form that is not enabled by other instruments.

Video-based categorization system and frequency analysis of gestures in saxophone playing

The study of gestures in music performance provides valuable insights for instrumental learning. However, gestural vocabularies vary depending on the instrument being played, according to its postural and technical specificities. The goals of this study were twofold: first, to create a gesture categorization system for saxophone players, and second, to analyse their gestural behaviour across contrasting musical excerpts. A criteria-based observational analysis was conducted, considering the type and frequency of gestures identified in a database of 100 video and motion recordings. The categorization system, including 15 gesture types applicable to the case of saxophone playing, was further validated by 2 expert raters. A descriptive appendix is provided for the identification of each gesture type. Results revealed that: (1) knee and trunk flexion, feet elevation, mediolateral sway and flap were the most recurrent gestures among saxophone players; (2) energetic, fast-tempo excerpts led to higher movement frequency; and (3) impulsive gestures (head nods) were idiosyncratic of the excerpt containing repeated accentuated notes. These results present a definition of the gestural behaviour of saxophone players, which constitutes relevant knowledge for the development of future studies in the fields of injury prevention, body expression and historically informed performance.

Plato’s Parasite: Re:turning with(in) Deleuze

ABSTRACT Upon encountering Deleuze’s Nietzschean reiteration that it is the task of modern philosophy to overturn Platonism, as put forth in Difference and Repetition (1968), this article worms its way in, seeking to unearth not merely what Deleuze’s differential repetition signifies, but also what such gestures generate in (re)turn. By considering Deleuze’s intervention as a manifold performance of (re)purposeful gestures that put the notion of reversal itself into question, and after providing a brief, Deleuzian, exposition of Platonism, this article not only homes in on Deleuze’s demiurgic detection of an overturning gesture paradoxically already present in, if not itself initiated by, Plato’s writing, but additionally calls attention to other curiosities turn up when reading besides, instead of against, the grain. In introducing Michel Serres as a critical companion, this experimental article ultimately ponders over and entertains the productive possibility of framing Plato’s poignant predicament as a (problem of the) parasite. Composed as fruitful conceptual personae, parasites emerge throughout this article in unforeseen manners, time and time again bringing new food for thought to the table.

Recovery of gestures for persons with severe non-fluent aphasia and limb apraxia: A long-term follow-up study

Persons with severe non-fluent aphasia would benefit from using gestures to substitute for their absent powers of speech. The use of gestures, however, is challenging for persons with aphasia and concomitant limb apraxia. Research on the long-term recovery of gestures is scant, and it is unclear whether gesture performance can show recovery over time. This study evaluated the recovery of emblems and tool use pantomimes of persons with severe non-fluent aphasia and limb apraxia after a left hemisphere stroke. The Florida Apraxia Screening Test-Revised (FAST-R) was used for measurements. The test includes 30 gestures to be performed (i) after an oral request, (ii) with the aid of a pictorial cue, or (iii) as an imitation. The gestures were rated on their degree of comprehensibility. The comprehensibility of gestures after an oral request improved significantly in five out of seven participants between the first (1–3 months after the stroke) and the last (3 years after) examination. Improvement continued for all five in the period between six months and three years. The imitation model did improve the comprehensibility of gestures for all participants, whereas the pictorial cue did so just slightly. The skill of producing gestures can improve even in the late phase post-stroke. Because of this potential, we suggest that gesture training should be systematically included in the rehabilitation of communication for persons with severe non-fluent aphasia.

Research on the Application of IU-EKF Algorithm for Feature Extraction of Flute Performers Based on Deep Learning

In order to achieve a more standardized teaching of flute playing gestures, a deep learning based application research on flute player hand shape feature extraction was proposed. Based on the theory of deep learning, a fusion network model based on VGG-16 was designed. The IU-EKF algorithm for deep learning was used to model, pose, learn, and train the hand shape of flute players. Experimental data analysis was conducted on the learning algorithm, and the experimental results showed that the model can extract gesture features for flute performance, whether on the back of the hand. The various angles of the upper and lower joints of the fingers have higher precision recognition than traditional methods. The feature extraction of flute gesture performance can also more effectively recognize the changes in different pitch angles of the gesture at different times, which has important reference value for the standardization of flute hand performance.

Gestural Alignment in Spoken Simultaneous Interpreting: A Mixed-Methods Approach

Cognitive and behavioral alignment plays a major role in simultaneous interpreting, the interpreter centrally monitoring and accommodating his/her behavior to that of the speaker-source. In parallel, the place of gesture in the interpreters’ practice, as well as its degree of convergence with respect to the gestures of the speaker-source, has been scarcely analyzed until very recently. The multimodal data for this study were collected under (quasi-)experimental conditions in a real court interpreting setting during spoken training exercises performed by two novice interpreters. In this exploratory study, the gestural performance of the interpreters, including their degree of gestural alignment towards the speaker-source, is analyzed and compared using a mixed-methods approach to a randomized sample of the recorded data. The analysis combines a basic descriptive quantification of body movements and a qualitative and comparative analysis of the gesture types performed by the speaker-source and the interpreters. The results show that, in spite of individual differences in interpreting fluency and gestural styles, both interpreters tend to align with the speaker-source’s gestural behavior in several ways, and thus a basic taxonomy of gestural convergence between the speaker-source and the interpreters is defined according to several criteria (mainly, gesture presence and gesture type). Our conclusions also allow us to formulate new research questions and hypotheses to be tested in future studies (e.g., types of gestures by the speaker-source that prompt a higher degree of alignment).

Research on fastDTW and isolation forest-based lightweight gesture authentication

This study explores the use of the accelerometer and gyroscope sensors in an Android mobile phone to capture data during user gesture performance. The recorded data is then processed and analyzed to extract feature values. The authentication of the user's identity may be achieved by this method, which is characterized by minimal constraints. This study initially compares the user's gesture to the template gesture using FastDTW. After that, an adaptive weighted vote determines the closest gesture template. This checks if the user is doing the template category correctly. The second stage extracts effective feature values from authentic user gestures. These feature values include efficient time-frequency domain feature values and the data extreme point spacing-to-length ratio. After that, authentic gestures are used to build the Isolation Forest to verify the user. This approach suggests a reduction in FRR, FAR, and an increase in accuracy, while using a smaller amount of data.

Recurrent gestures and embodied stance-taking in courtroom opening statements

Abstract Despite the inherently multimodal nature of the courtroom, studies of multimodality in forensic linguistics have been scarce. This study uses the stance triangle and ideological square concepts and the assumption that recurrent gestures serve as stance-taking resources to analyze conflicting embodied stances taken by the prosecutor and defense attorney in their opening remarks during the State of Minnesota v. Derek Michael Chauvin 2021 trial. Statistical results reveal that the prosecutor predominantly uses modal gestures to display a strongly aggravated oppositional stance and attempts to be more persuasive, while the defense attorney favors performative gestures. The analysis of the interplay between the verbal and gestural resources suggests that recurrent gestures such as open hand prone or supine, open hand held with vertical palms, index finger extended, and precision grip have a threefold indexical realization: prototypical, argumentative, and stance-taking. The findings contribute to a more complete understanding of persuasion in the courtroom.

Fashion as a cultural analysis object

I reflect upon working on diverse fashion(-related) cases in the context of the Amsterdam School for Cultural Analysis. Drawing on Mieke Bal’s understanding of cultural analysis (characterized by interdisciplinarity, among others), I refer to some fashion and material objects I have looked at (e.g. a contemporary video by a Dutch designer, a wardrobe-suitcase of 1919 or the pandemic mask), while insisting on considering them as theoretical objects through the practice of clothes-reading. Rather than establishing programmatic grids to study fashion, I share my observations and interrogations on working with and through fashion. What follows resembles an ongoing inventory of my fashion interests, aspirations and inspirations as well as choices and doubts that altogether echo the performative gesture of research.

Gesture profiles distinguish primary progressive aphasia variants: a preliminary study

ABSTRACT Background Primary progressive aphasia (PPA) is a neurodegenerative syndrome characterized by progressive language deficits. The main variants of PPA – semantic (svPPA), logopenic (lvPPA), and nonfluent (nfvPPA) – can be challenging to distinguish. Limb apraxia often co-occurs with PPA, but it is unclear whether PPA variants are associated with different gesture deficits. Prior evidence from stroke indicates that temporal lobe lesions are associated with reduced benefit from meaning in gesture performance. Temporal lesions are also associated with greater deficits in hand postures compared to arm kinematics in gesture imitation. Aims We tested the hypothesis that svPPA, who primarily exhibit temporal lobe atrophy, would differentially show this pattern. In addition, we predicted that as a group, individuals with PPA would more accurately produce meaningful than meaningless gestures and kinematic components than hand postures, as found in neurotypical adults and individuals with stroke. Methods & Procedures Participants with PPA completed meaningful and meaningless gesture imitation tasks. Performance was scored for hand posture and arm kinematic components. Bayesian mixed-effect models examined trial-level gesture imitation accuracy, while controlling for dementia severity and random effects of subjects and items. Fixed effects included an interaction between PPA variant, task condition/meaning, and gesture component. Outcomes & Results We found a critical interaction between PPA variant, meaning, and gesture component. In particular, unlike lvPPA and nfvPPA, svPPA subjects’ hand postures failed to benefit from gesture meaning. This preliminary research extends prior findings on the role of the temporal lobe in action representations associated with manipulable objects, and is among the first to demonstrate distinct gesture imitation patterns between PPA variants. Conclusions Characterizing gesture deficits in PPA may help inform diagnostics, compensatory communication strategies, and models of praxis.

Assistance by adaptative damping on a complex bimanual task in laparoscopic surgery.

Laparoscopic surgery has improved outcomes in abdominal surgery, but presents kinematic restrictions for surgeons. Robotic comanipulation with adaptative damping has been investigated in simple laparoscopic tasks. The present protocol aimed to determine the contribution of adaptive damping in complex bimanual tasks approaching clinical setting. Fourteen residents in general surgery performed three exercises, and for each three repetitions without (classic repetitions) and three with robotic assistance (robotic repetitions) in a randomised order. The exercises chosen were trajectory, modified Pea on a Peg and intracorporeal suture. Task performance, gesture performance, workload and impression were measured. Also, a semi-directed interview was performed to collect the participants' feeling about companipulated robots and their potential application in clinical practice. Adaptative damping assistance did not impact task performance, but allowed an economy of movement in the non-dominant hand during suture exercise (distance 916 ± 500mm in classic vs. 563 ± 261mm in robotic, p < 0.001). Perceived workload (p = 0.12) and user's impression were not different between classic and robotic repetitions, except novelty (p < 0.001). Participants' interviews revealed their interest for the robotic devices, particularly the gravity compensation, and were ready to use the adaptative damping provided an intermittent use, for example to dissect dangerous areas. Adaptative damping applied by comanipulated robots does not influence the performance of the task, but improves the performance of the gesture itself, particularly for the non-dominant hand, and during the realisation of a complex task like suturing. For residents in digestive surgery, this assistance does increase workload, and they would use this help in the operating room under certain conditions.

The characteristics of music video experiences and their relationship to future listening outcomes

Music videos are a popular method of consuming music; however, the characteristics of these experiences and their effects on music perception are poorly understood. An online survey ( N = 155) was designed using theoretical insight from Dasovich-Wilson et al.’s (2022) Intention Attention Reaction and Retention (IARR) framework. The survey consisted of two parts: the first explored the key characteristics of music video experiences, and the second explored their effects on subsequent listening outcomes. Separate principal component analyses (PCAs) were performed on each part to differentiate between the experience itself (Experience components) and the effects on subsequent listens (Retention outcomes). Relationships between Experience components and Retention outcomes were explored using correlation and regression analyses. The results suggest that music video experiences characterized by performance gestures and narratives have the strongest influence on music perception. These findings shed light on how extramusical information from music videos influences mechanisms related to visual imagery and personal associations.

In Search of the “Phonograph Effect”: Sonic Gestures in Violin Performance and Their Modification by Early Recording and Playback Devices (1901–1933)

To what extent did early recording technology affect the creation and representation of musical performances? According to Mark Katz (from 1999 onwards), historical studio environments led to crucial shifts in 20th century violin performances due to the restrictions imposed by early recording and reproduction devices (“phonograph effects”). In particular, this may have affected sonic gestures that include expressive means such as vibrato, portamento, articulation, and timbre variation. In order to trace potential modifications, we reenacted a 1911 “Liebesleid” performance by one of the most influential violinists of the 20th century, Fritz Kreisler. We then digitally ascertained the full acoustic transfer paths (impulse responses, IRs) from the 1911 studio to 20 historical gramophone setups and applied them to the reenactment. In this way, for the first time, our study generated comparative IR findings across multiple gramophones, soundboxes, and horns built by different manufacturers between 1901 and 1933. Sonic gestures were found to induce significant level modifications of up to 20 dB due to the devices’ resonances, leading to dynamical variations that have never been part of the performance. Accordingly, Kreisler's famous “golden tone” is due, in part, to the recording technologies of his time. Therefore, early recordings should not be understood as “neutral witnesses” but rather as artifacts with substantial influence on the creation and reproduction of musical performance(s).

Gestures as Scaffolding to Learn Vocabulary in a Foreign Language.

This paper investigates the influence of gestures on foreign language (FL) vocabulary learning. In this work, we first address the state of the art in the field and then delve into the research conducted in our lab (three experiments already published) in order to finally offer a unified theoretical interpretation of the role of gestures in FL vocabulary learning. In Experiments 1 and 2, we examined the impact of gestures on noun and verb learning. The results revealed that participants exhibited better learning outcomes when FL words were accompanied by congruent gestures compared to those from the no-gesture condition. Conversely, when meaningless or incongruent gestures were presented alongside new FL words, gestures had a detrimental effect on the learning process. Secondly, we addressed the question of whether or not individuals need to physically perform the gestures themselves to observe the effects of gestures on vocabulary learning (Experiment 3). Results indicated that congruent gestures improved FL word recall when learners only observed the instructor's gestures ("see" group) and when they mimicked them ("do" group). Importantly, the adverse effect associated with incongruent gestures was reduced in the "do" compared to that in the "see" experimental group. These findings suggest that iconic gestures can serve as an effective tool for learning vocabulary in an FL, particularly when the gestures align with the meaning of the words. Furthermore, the active performance of gestures helps counteract the negative effects associated with inconsistencies between gestures and word meanings. Consequently, if a choice must be made, an FL learning strategy in which learners acquire words while making gestures congruent with their meaning would be highly desirable.

The Use of Thread and Fabric in Feminist Zines

The Use of Thread and Fabric in Feminist Zines

Touching Disney: Loneliness and Sources of Comfort in the Digital Age

Reflecting on moving image affect and human-computer interaction, the following paper considers the connections between elective states of loneliness and different spectatorship habits, including excessive media consumption and escapism into nostalgic media texts. Comparing the experience of communal spectatorship fostered by the space of the cinema in the early twentieth century to the accessibility and endless choice of moving image content facilitated by streaming platforms, it also considers the cultural value and the status of moving image as a manifestation in the world. Drawing on examples from streaming platforms' content, and especially the model provided by Disney Plus, the following exploration illustrates the transformation of familiar media texts into emotionally charged objects of nostalgia, agents of deeply rooted childhood memories, and often, sources of comfort. Their presence on handheld devices facilitates new forms of engagement, including repetitive viewing, tactility and increased levels of intimacy and interactivity through the nature of touchscreens, including holding, scrolling, pinching, swiping and other performative gestures. Drawing on existing definitions of loneliness and emerging theories of audience engagement with touch-screen technologies, this paper considers practices of repetitive watching and binge-watching, new forms of viewing familiar content, as well as the spectators' historic relationship to Disney content, as responses to loneliness triggered by emotional or social isolation.

Remember Me: Vocal Cadenzas as Mementos in Nineteenth-Century Musical Autograph Albums

Abstract Between the 1830s and the 1880s, leading female singers such as Henriette Rossi Sontag, Laure Cinti-Damoreau, Adelina Patti, and Pauline Viardot participated in a widespread custom of inscribing original vocal cadenzas in autograph albums (also known as friendship albums, keepsake albums, alba amicorum, or Stammbücher). The appeal of album cadenzas, along with other compact inscriptions, reflects a wider Romantic fascination with remembering through small keepsakes such as miniature almanacs and annual compilations of poetry, prose, or music. Whether improvised in the presence of collectors or offered as souvenirs of earlier performances, cadenzas functioned as mementos of musical and social encounters while visually representing singers’ musical personae through performative gestures. These florid pieces also allowed female singers to exercise agency as both creative artists and curators of their public images, much like the (mostly male) composers whose musical signatures fill surrounding album leaves. As performance-practical sources, vocal cadenzas preserve rare and revealing traces of singers’ artistry; many, in fact, can be linked to specific concerts or performances of operas by Bellini, Donizetti, Meyerbeer, Rossini, Verdi, and others. In the case of Cinti-Damoreau, such music illuminates the connection between improvisation, performance, and pedagogy. The gradual disappearance of album cadenzas during the last quarter of the nineteenth century is attributable not only to a waning interest in musical improvisation generally, but also to a shift in creative authority away from singers and toward composers, particularly in the operatic realm.

Snapture—a Novel Neural Architecture for Combined Static and Dynamic Hand Gesture Recognition

As robots are expected to get more involved in people’s everyday lives, frameworks that enable intuitive user interfaces are in demand. Hand gesture recognition systems provide a natural way of communication and, thus, are an integral part of seamless human-robot interaction (HRI). Recent years have witnessed an immense evolution of computational models powered by deep learning. However, state-of-the-art models fall short of expanding across different gesture domains, such as emblems and co-speech. In this paper, we propose a novel hybrid hand gesture recognition system. Our Snapture architecture enables learning both static and dynamic gestures: by capturing a so-called snapshot of the gesture performance at its peak, we integrate the hand pose and the dynamic movement. Moreover, we present a method for analyzing the motion profile of a gesture to uncover its dynamic characteristics, which allows regulating a static channel based on the amount of motion. Our evaluation demonstrates the superiority of our approach on two gesture benchmarks compared to a state-of-the-art CNNLSTM baseline. Our analysis on a gesture class basis unveils the potential of our Snapture architecture for performance improvements using RGB data. Thanks to its modular implementation, our framework allows the integration of other multimodal data, like facial expressions and head tracking, which are essential cues in HRI scenarios, into one architecture. Thus, our work contributes both to integrative gesture recognition research and machine learning applications for non-verbal communication with robots.

Cross-Modal Musical Expectancy in Complex Sound Music: A Grounded Theory.

Expectancy is a core mechanism for constructing affective and cognitive experiences of music. However, research on musical expectations has been largely founded upon the perception of tonal music. Therefore, it is still to be determined how this mechanism explains the cognition of sound-based acoustic and electroacoustic music, such as complex sound music (CSM). Additionally, the dominant methodologies have consisted of well-controlled experimental designs with low ecological validity that have overlooked the listening experience as described by the listeners. This paper presents results concerning musical expectancy from a qualitative research project that investigated the listening experiences of 15 participants accustomed to CSM listening. Corbin and Strauss' (2015) grounded theory was used to triangulate data from interviews along with musical analyses of the pieces chosen by the participants to describe their listening experiences. Cross-modal musical expectancy (CMME) emerged from the data as a subcategory that explained prediction through the interaction of multimodal elements beyond just the acoustic properties of music. The results led to hypothesise that multimodal information coming from sounds, performance gestures, and indexical, iconic, and conceptual associations re-enact cross-modal schemata and episodic memories where real and imagined sounds, objects, actions, and narratives interrelate to give rise to CMME processes. This construct emphasises the effect of CSM's subversive acoustic features and performance practices on the listening experience. Further, it reveals the multiplicity of factors involved in musical expectancy, such as cultural values, subjective musical and non-musical experiences, music structure, listening situation, and psychological mechanisms. Following these ideas, CMME is conceived as a grounded cognition process.