Motion Gesture Research Articles

Human motion and musical expression: Exploring the link between physical movement and emotional resonance in performance

Musical performance has traditionally been understood as an auditory experience, with emotions conveyed primarily through sound. However, emerging research highlights the role of physical movement—gestures, posture, and body language—in enhancing emotional expression and creating a deeper connection between performers and their audiences. While existing studies have explored some aspects of this relationship, there is limited understanding of how specific movements modulate emotional responses across different musical genres. This study addresses these gaps by investigating the interaction between human motion and musical expression, focusing on the emotional resonance experienced by performers and audience members. The need for this study arises from the lack of comprehensive data on the correlation between movement and emotional engagement during musical performances, particularly concerning different genres. Using motion capture, biometric sensors, and facial recognition technology, the research analyzed the performances of professional and amateur musicians across classical, jazz, and contemporary music. Audience members’ emotional responses were captured through physiological data and post-performance surveys. One limitation of this study is that it focused solely on live performance environments, leaving digital or virtual performances unexplored. Key findings reveal that expressive movements increased emotional intensity ratings by 1.6 points (p < 0.001), with heart rate rising by 6.9 BPM (p < 0.0012) and electrodermal activity increasing by 1.6 µS (p < 0.0008). Movement synchronization with musical elements, especially during climaxes, strongly correlated with heightened emotional responses (r = 0.85, p < 0.01). Classical performances exhibited the highest synchronization between movement and emotion, with audience emotional intensity peaking at 8.3 on a Likert scale.

Embodiment of sexuality by Israeli Sign Language signers and hearing Hebrew speakers

AbstractLanguage is a key resource for speakers and signers to index different aspects of their social identities, such as their ethnicities and sexualities. Yet, for users of sign language – who exploit movements of the hands, face, head and torso for linguistic purposes – it is often assumed that any communicative movement of the body is part of a sign language rather than a general feature of the body's potential to communicate social meaning, shared by hearing and deaf individuals. In this study, we test this claim by comparing the movement features produced by gay and straight Israeli Sign Language signers to the gestural movements produced by gay and straight Hebrew speakers. The findings reveal that deaf gay signers and hearing gay gesturers exploit similar movements of the body. By incorporating the notion of embodiment into sign language sociolinguistics, we can better conceptualize the relationship between sign language and social identities.

Exploring 2020 #ENDSARS Protests’ Components in Nigeria within the Schematics of Social Semiotics

This study examined the semiotic resources of the October 2020 #ENDSARS protests in Nigeria, manifesting the affordances operational in the protests. The investigation beckoned on 13 different communicative semiotic configurations, employing Halliday’s grammatical group rank, to delineate the structures. This paved the way for tables and graphs to account for the written modes’ recurrent components. Kinesics such as proxemics, gestures, and gaze supported the analysis, providing relevant semantic implications to non-verbal elements and body language of protesters. This investigation indicated that #ENDSARS hashtag, functioned in variegated forms, as the initiator of the protest, socializing people even without notable leaders. The texts illustrated the suffering of the people from SARS operatives’ gun-punch coercion, as exemplified in a boy-like image’s head, and the cry of agony illuminated through a gestural movement of mouths and hands. Parallel to that is the metaphorical castigation of an innocent young man, manifested through chained hands. Significantly, the research revealed the collective unanimity of the protesters, demonstrated through clenched fists, epitomizing camaraderie of purpose. The protesters made their call to abrogate SARS operations pleasurably-funny in the appearance of a clown and a sacrificial lamb. Given that the idea of social justice is non-negotiable. Demanding a just course should not lead to brutality; the government should rather respond to the request of protesters to fulfill promises made during electioneering campaigns.

Do speakers’ gestures affect listeners’ understanding of temporal relationships between events?

Abstract This study investigated how the direction and movement of gestures affect mental representations of time. Previous research by Tversky and Jamalian (2021) demonstrated that gestures influence how people perceive temporal relationships, with participants’ drawings reflecting the shape (circular or linear) of the gestures they observed. However, the influence of gesture direction was not examined. In the present study, 50 Japanese adults were exposed to different types of gestures, including leftward and rightward linear or circular gestures. Participants were then asked to draw diagrams representing the temporal flow of events. The results showed significant associations between the type of gesture and the participants’ drawings. Specifically, the direction and form of the gestures — whether leftward or rightward, linear or circular — significantly influenced how participants depicted temporal relationships. These findings highlight the importance of both gesture direction and movement in shaping individuals’ mental representations of time.

A study of gestures representing musical emotions

The creation of music has originated from the burst of inspiration of the composer. Many studies suggested that improvisation plays a more vital role than composition in the emergence of new forms of music. The production of music, especially in Western cultures, tends to follow its traditional music theory. With the emergence of Pythagorean music theory, Western musicology has been defined and developed in depth. The composer’s creation is based on music creation within the framework of the Western music theory. However, in recent years, with the rise of the topic of music gestures, some scholars have learned that music ontology as a four-dimensional hypercube is inseparable from the performer’s gesture. Therefore, some musicians also attempt to create flexible music from the perspective of music performers and based on performance gestures. This kind of music creation breaks the traditional music theory system and has more personal style characteristics, but few people can understand the thinking logic behind this kind of creative behavior. Therefore, this article focuses on the singing gesture and reversely analyzes its impact on musical creation. We use a recent mathematic definition of gestures and we discuss emotion from its etymological origin (from the Latin e-movere) as a gestural movement from inside out. This article conducts an in-depth analysis based on the singer’s gesture for the following two reasons: First, the literature on the connection between singing gesture theory and composition theory remains scarce. Second, gestures produced by a singer are the only performance form that is not enabled by other instruments.

Highly durable yarn-based strain sensor with enhanced underwater monitoring capabilities and rapid drowning detection for safety applications

Flexible strain sensors have garnered significant attention for their outstanding performance in fields such as telemedicine, motion management, and human–machine interfaces. However, in the demanding marine environments, these advanced sensors face challenges related to mechanical weakness, stability, and durability under wet conditions, which are critical for expanding their range of practical applications. In this study, a stable and durable yarn sensor was developed specifically tailored for underwater activity monitoring. This was achieved by using polydopamine to immobilize silver nanoparticles, thereby creating a highly conductive layer, followed by applying an impregnation coating to establish a hydrophobic layer. The resulting composite structure featured a micro-convex layer of silver nanoparticles, which provided a durable hydrophobic surface resistant to mechanical wear and capable of maintaining its electrical properties even under high-velocity water impacts. More significantly, a FWAS was engineered, which is activated by pre-stretching the yarn sensor using a water-soluble vinylon yarn. The FWAS automatically alters its resistance within 1.3 s upon detecting someone falling into the water, thereby sending out an accurate distress signal to the external world by transmitting Morse code signals via gesture movements. This innovation provides a straightforward and effective approach for timely rescue operations from drowning, demonstrating the versatility of yarn-based flexible strain sensors for use not only in air but also in underwater scenarios.

Nanocellulose and multi-walled carbon nanotubes reinforced polyacrylamide/sodium alginate conductive hydrogel as flexible sensor

Conductive hydrogels have been widely applied in human–computer interaction, tactile sensing, and sustainable green energy harvesting. Herein, a double cross-linked network composite hydrogel (MWCNTs/CNWs/PAM/SA) by constructing dual enhancers acting together with PAM/SA was constructed. By systematically optimizing the compositions, the hydrogel displayed features advantages of good mechanical adaptability, high conductivity sensitivity (GF = 5.65, 53 ms), low hysteresis (<11 %), and shape memory of water molecules and temperature. The nanocellulose crystals (CNWs) were bent and entangled with the backbone of the polyacrylamide/ sodium alginate (PAM/SA) hydrogel network, which effectively transferred the external mechanical forces to the entire physical and chemical cross-linking domains. Multi-walled carbon nanotubes (MWCNTs) were filled into the cross-linking network of the hydrogel to enhance the conductivity of the hydrogel effectively. Notably, hydrogels are designed as flexible tactile sensors that can accurately recognize and monitor electrical signals from different gesture movements and temperature changes. It was also assembled as a friction nanogenerator (TENG) that continuously generates a stable open circuit voltage (28 V) for self-powered small electronic devices. This research provides a new prospect for designing nanocellulose and MWCNTs reinforced conductive hydrogels via a facile method.

Sensitive strain sensors with wide-range detection and high stability for intelligent human motion monitoring

Miniaturized wearable electronic devices with strain sensors as core sensing elements hold significant applications in health monitoring and human-machine interaction. However, it remains a challenge to develop strain sensors with optimal response linearity, a wide detection range, and stability. In this study, to satisfy the above requirements, we construct high-performance wearable strain sensors (rGO-CNTOH@PSE) by a strategy of hybridizing conductive fillers with stretchable substrates. In detail, a synergistic conductive network is developed by adopting OH-containing carbon nanotube (CNTOH) and reduced graphene oxide (rGO) fillers with an appropriate mass ratio. After the network is integrated into a porous silicone elastomer substrate formed by the salt templating method, the developed strain sensor exhibits distinct resolution, consistent repeatability, and superior response linearity across a wide strain detection range of 0.3–303.03 %. Furthermore, the sensor also maintains good response stability over 15000 cycles of strain. We also demonstrate that this strain sensor showing outstanding exceptional detection performance has the capability for full-range human motion detection. A wireless smart electronic glove is developed based on the sensor, which can accurately recognize gesture movements and control a bionic robotic hand to replicate human hand actions, offering the potential for replacing manual tasks (such as, grasping in hazardous environments) in the future.

Application of digital entertainment experience based on intelligent interactive system in personalized fitness training system

The application of digital media in fitness environments can improve the entertainment of the fitness process, allowing users to watch entertainment videos while engaging in fitness activities. At the same time, we can add relevant game elements to the fitness process, such as playing tennis through gesture movements, thereby increasing user interest in fitness participation. Therefore, this article has developed a personalized fitness training system using optical sensors and intelligent algorithms, and conducted testing experiments on blood oxygen saturation under two sports states: running and cycling. This article analyzes the intelligent optimization algorithm for optical signals. This algorithm has strong adaptability and intelligence level, and can directly learn and adjust the model based on the characteristics of the data. The automation performance of the intelligent algorithm can effectively reduce labor costs and improve efficiency. Finally, this article provides a basic analysis and testing of the personalized fitness training system. Through the obtained data, personalized guidance is provided to users, helping people make fitness more scientific and intelligent, and improving fitness efficiency.

Scope of Negation, Gestures, and Prosody: The English Negative Quantifier as a Case in Point.

The present paper examines how English native speakers produce scopally ambiguous sentences and how they make use of gestures and prosody for disambiguation. As a case in point, the participants in the present study produced the English negative quantifiers. They appear in two different positions as (1) The election of no candidate was a surprise (a: 'for those elected, none of them was a surprise'; b: 'no candidate was elected, and that was a surprise') and (2) no candidate's election was a surprise (a: 'for those elected, none of them was a surprise'; b: # 'no candidate was elected, and that was a surprise.' We were able to investigate the gesture production and the prosodic patterns of the positional effects (i.e., a-interpretation is available at two different positions in 1 and 2) and the interpretation effects (i.e., two different interpretations are available in the same position in 1). We discovered that the participants tended to launch more head shakes in the (a) interpretation despites the different positions, but more head nod/beat in the (b) interpretation. While there is not a difference in prosody of no in (a) and (b) interpretation in (1), there are pitch and durational differences between (a) interpretations in (1) and (2). This study points out the abstract similarities across languages such as Catalan and Spanish (Prieto et al. in Lingua 131:136-150, 2013. 10.1016/j.lingua.2013.02.008; Tubau et al. in Linguist Rev 32(1):115-142, 2015. 10.1515/tlr-2014-0016) in the gestural movements, and the meaning is crucial for gesture patterns. We emphasize that gesture patterns disambiguate ambiguous interpretation when prosody cannot do so.

Evaluating Gesture Generation in a Large-scale Open Challenge: The GENEA Challenge 2022

This article reports on the second GENEA Challenge to benchmark data-driven automatic co-speech gesture generation. Participating teams used the same speech and motion dataset to build gesture-generation systems. Motion generated by all these systems was rendered to video using a standardised visualisation pipeline and evaluated in several large, crowdsourced user studies. Unlike when comparing different research articles, differences in results are here only due to differences between methods, enabling direct comparison between systems. The dataset was based on 18 hours of full-body motion capture, including fingers, of different persons engaging in a dyadic conversation. Ten teams participated in the challenge across two tiers: full-body and upper-body gesticulation. For each tier, we evaluated both the human-likeness of the gesture motion and its appropriateness for the specific speech signal. Our evaluations decouple human-likeness from gesture appropriateness, which has been a difficult problem in the field. The evaluation results show some synthetic gesture conditions being rated as significantly more human-like than 3D human motion capture. To the best of our knowledge, this has not been demonstrated before. On the other hand, all synthetic motion is found to be vastly less appropriate for the speech than the original motion-capture recordings. We also find that conventional objective metrics do not correlate well with subjective human-likeness ratings in this large evaluation. The one exception is the Fréchet gesture distance (FGD), which achieves a Kendall’s tau rank correlation of around -0.5. Based on the challenge results we formulate numerous recommendations for system building and evaluation.

Hand Gesture Recognition Based on High-Density Myoelectricity in Forearm Flexors in Humans.

Electromyography-based gesture recognition has become a challenging problem in the decoding of fine hand movements. Recent research has focused on improving the accuracy of gesture recognition by increasing the complexity of network models. However, training a complex model necessitates a significant amount of data, thereby escalating both user burden and computational costs. Moreover, owing to the considerable variability of surface electromyography (sEMG) signals across different users, conventional machine learning approaches reliant on a single feature fail to meet the demand for precise gesture recognition tailored to individual users. Therefore, to solve the problems of large computational cost and poor cross-user pattern recognition performance, we propose a feature selection method that combines mutual information, principal component analysis and the Pearson correlation coefficient (MPP). This method can filter out the optimal subset of features that match a specific user while combining with an SVM classifier to accurately and efficiently recognize the user's gesture movements. To validate the effectiveness of the above method, we designed an experiment including five gesture actions. The experimental results show that compared to the classification accuracy obtained using a single feature, we achieved an improvement of about 5% with the optimally selected feature as the input to any of the classifiers. This study provides an effective guarantee for user-specific fine hand movement decoding based on sEMG signals.

Application of Fashion Element Trend Prediction Model Integrating AM and EfficientNet-b7 Models in Art Design

The poor application effect of traditional sports training methods and the difficulty of recording data due to the time and space constraints of sports make it difficult for trainers to improve their learning outcomes. Based on this, the study proposes to apply human posture recognition in sports teaching design, and use VGGNet-19 as a feature extractor and OpenCV open source software to capture posture movements, and introduce the concepts of joint angle and movement similarity to design a sports assessment system for physical education based on the geometric spatial feature variability analysis of posture based on limb angle information. The testing outcomes demonstrate that the study’s improved gesture recognition algorithm has a recognition rate of more than 90% on gesture movements, and the maximum recognition error value (0.010) is smaller than that of the dynamic time-regularised gesture algorithm (0.014) and the convolutional neural network algorithm (0.017). The assessment system is also better able to improve students’ professional performance and satisfaction, with its average professional score and satisfaction reaching 86 and 92%, which is significantly better than other comparative algorithms. The method is effective in providing trainers with data-based training scenarios and helping them to improve their learning in sport.

Temporal Action Segmentation in Sign Language System for Bahasa Indonesia (SIBI) Videos Using Optical Flow-Based Approach

Sign language (SL) is vital in fostering communication for the deaf and hard-of-hearing communities. Continuous Sign Language Translation (CSLT) is a work that translates sign language into spoken language. CSLT translation is done by changing continuous forms into isolated signs. Segmenting morpheme signs from phrase signs has several challenges, such as the availability of annotated datasets and the complexity of continuous gesture movements. The Indonesian Sign Language (SIBI) system follows Indonesian grammatical norms, including word formation, in contrast to other sign languages with rules derived from their spoken language. In SIBI, a word can consist of a root word and an affix word. Therefore, temporal action segmentation in SIBI is important to reconstruct the results of translating each sign into spoken Indonesian sentences. This research uses an optical flow approach to segment temporal actions in SIBI videos. Optical flow methods that calculate changes in intensity between adjacent frames can be used to determine the occurrence of sign movement or vice versa to determine the delay between sign movements. The absence of intensity differences between the two frames indicates the boundary between sign gestures. This study tested the use of dense optical flow on videos containing SIBI sentences taken from 3 signers. Evaluation is done on several parameters in the dense optical flow algorithm, such as threshold size, PyrScale, and WinSize, to obtain the best accuracy. This paper shows that the optical flow algorithm successfully performs segmentation, as measured by Perf and F1r. The experimental results showed that the highest Perf and F1r yields were 0.8298 and 0.8524, respectively.

Development of a Motion Controller for the Electric Wheelchair of Quadriplegic Patients Using Head Movements Recognition

The robotic wheelchair is designed to allow independent movement for patients with severe musculoskeletal and neuromuscular disorders that interfere with normal wheelchair propulsion by hands. Control of the movement of the wheelchair necessities residual motion of the upper extremity which was not applicable for patients with upper extremity disability as in Quadriplegic patients. Several research efforts have been proposed to help quadriplegic patients independently use the robotic wheelchair. However, the proposed solutions had many problems with accuracy, simplicity, and cost. The current work aims to build an accurate, simple, and low-cost control system for robotic wheelchairs using head movement. The proposed system was based on the use of the MEMs Inertial measurement unit (IMU) to sense the head movement gestures and transfer the gestures into signals to control the wheelchair movements. A new lightweight algorithm for the detection of head gestures was proposed to improve the system’s accuracy. The proposed system has been applied and tested on volunteers. The results revealed that the accuracy of the proposed method in recognizing head movement was 97% on average.

Trajectory Tracking Algorithm Study of Coal Mine Water Detector Drilling Bar Installation

Mechanical water detection is recognized as the most reliable and safe production technology for coal mines, mainly for the detection of water hazards in pre-mining operations. Intelligent water detectors are currently the main research direction in mechanical water detection, and the automatic installation of drilling bars is the key to achieving intelligent water detection. Improving the connection accuracy in the process of installing drilling bars is an important research topic for the improvement of control links. To improve the connection accuracy of the drilling bars at the time of supplying material, we used the modified Denavit–Hartenberg method to analyze the motion gestures of the supplied material device and the Lagrange equation to establish a dynamic analysis model. We aimed at better control precision by improving the sliding mode control algorithm and at increasing the convergence rate of tracking errors with a sliding controller based on an exponential approximation law and using saturated functions instead of the symbol functions in the reaching law to weaken the vibration in the control process. We then used particle swarm optimization (PSO) to find the optimum combination parameters of the sliding mode controllers and test the performance of the sliding mode controllers before and after PSO with MATLAB/Simulink. The results showed that the optimized controller has a strong resistance to parameter fluctuations, and the system responds quickly, achieves a good performance, and improves the convergence rate of tracking errors.

Statement of Retraction: An Improved Approach for Human Movement Gesture Recognition Based on Grey Wolf Optimization and 3D Reconstruction

Statement of Retraction: An Improved Approach for Human Movement Gesture Recognition Based on Grey Wolf Optimization and 3D Reconstruction

Deep learning approach to improve the recognition of hand gesture with multi force variation using electromyography signal from amputees

Variations in muscular contraction are known to significantly impact the quality of the generated EMG signal and the output decision of a proposed classifier. This is an issue when the classifier is further implemented in prosthetic hand design. Therefore, this study aims to develop a deep learning classifier to improve the classification of hand motion gestures and investigate the effect of force variations on their accuracy on amputees. The contribution of this study showed that the resulting deep learning architecture based on DNN (deep neural network) could recognize the six gestures and robust against different force levels (18 combinations). Additionally, this study recommended several channels that most contribute to the classifier's accuracy. Also, the selected time domain features were used for a classifier to recognize 18 combinations of EMG signal patterns (6 gestures and three forces). The average accuracy of the proposed method (DNN) was also observed at 92.0 ± 6.1 %. Moreover, several other classifiers were used as comparisons, such as support vector machine (SVM), decision tree (DT), K-nearest neighbors, and Linear Discriminant Analysis (LDA). The increase in the mean accuracy of the proposed method compared to other conventional classifiers (SVM, DT, KNN, and LDA), was 17.86 %. Also, the study's implication stated that the proposed method should be applied to developing prosthetic hands for amputees that recognize multi-force gestures.

Research on Teaching Methods of Integrating Watercolor Techniques and Information Technology in Higher Education Art Education

Abstract Watercolor painting is a visual and creative discipline, which is closely related to the development of digital technology, which also affects the methods and means of basic art education in China. In this paper, we detect the trajectory of gesture movement after preprocessing the gesture data collected from students in watercolor painting using a Leap Motion sensor. We use Euclidean distance and nearest neighbor laws to identify and classify gesture movements in order to recognize students’ gesture movements in art education. Then, the virtual interactive system is designed with Unity 3D software, and finally, the art teaching method combining watercolor techniques and information technology is established. The results of teaching practice show that the level of students’ watercolor technique skills in all dimensions under innovative teaching is much higher than that of students under traditional teaching, and the significance values are all less than 0.05, which indicates that the use of VR technology has a very obvious impact on the teaching of watercolor painting. Meanwhile, the scores of students’ watercolor painting works were also improved with the assistance of VR technology. This study can provide a case reference for the subsequent research on the application of augmented reality technology and related teaching resources to art education and can enrich the construction of a new model of art learning activities in colleges and universities with technical support.

An investigation on the application of deep reinforcement learning in piano playing technique training

Abstract The article applies the knowledge of the physiological structure of finger movement in piano playing and the DH parameter method to construct a kinematic model of finger keystroke in piano playing. The Leap Motion-based gesture recognition algorithm is used for piano playing gesture movement extraction, gesture movement sub-framing, and velocity direction encoding, and the judgment HMM algorithm and the Viterbi improvement algorithm based on the a priori knowledge of fingering are utilized to automatically standardize piano fingering. After comparing the recognition performance of the piano playing training model in this paper, it is discussed how it can be used and how it can improve the user’s piano playing skills. In comparison to other piano gesture recognition models, the recognition accuracy and gesture dynamic information description of this paper’s training model demonstrate optimal performance. After the actual playing training experiments, the experimental group achieved significant improvements, while the control group experienced negligible improvements. After the experiment, the difference between the two groups increased, and the experimental group was superior to the control group in all aspects of piano playing. The training model for piano playing in this paper is effective in enhancing the user’s piano playing level.