Natural Human-computer Interaction Research Articles

A Novel Method for Hand Movement Recognition Based on Wavelet Packet Transform and Principal Component Analysis with Surface Electromyogram.

As an input method of signal language, the hand movement classification technology has developed into one of the ways of natural human-computer interaction. The surface electromyogram (sEMG) signal contains abundant human movement information and has significant advantages as the input signal of human-computer interaction. However, how to effectively extract components from sEMG signals to improve the accuracy of hand motion classification is a difficult problem. Therefore, this work proposes a novel method based on wavelet packet transform (WPT) and principal component analysis (PCA) to classify six kinds of hand motions. The method applies WPT to decompose the sEMG signal into multiple sub-band signals. To efficiently extract the intrinsic components of the sEMG signal, the classification performance of different wavelet packet basis functions is evaluated. The PCA algorithm is used to reduce the dimension of the feature space composed of the features reflecting hand motions extracted from each sub-band signal. Besides, to ensure higher classification performance while reducing the dimension of the feature space by the PCA algorithm, the classification performance of different dimensions of the feature space is compared. In addition, the effects of the variability of the sEMG signal and the size of the window on the proposed method are further analyzed. The proposed method was tested on the sEMG for Basic Hand Movements Data Set and achieved an average accuracy of 96.03%. Compared with the existing research, the proposed method has better classification performance, which indicates that the research results can be applied to the fields of exoskeleton robot, rehabilitation training, and intelligent prosthesis.

MmGesture: Semi-supervised gesture recognition system using mmWave radar

Gesture recognition has found versatile applications in natural human–computer interaction (HCI). Compared with traditional camera-based or wearable sensors-based solutions, gesture recognition using the millimeter wave (mmWave) radar has attracted growing attention for its characteristics of contact-free, privacy-preserving and less environment-dependence. Recently, most of studies adopted one of the Range Doppler Image (RDI), Range Angle Image (RAI), Doppler Angle Image (DAI) or Micro-Doppler Spectrogram extracted from the raw radar signal as the input of a deep neural network to realize gesture recognition. However, the effectiveness of these four inputs in gesture recognition has attracted little attention so far. Moreover, the lack of large amounts of labeled data restricts the performance of traditional supervised learning network. In this paper, we first conducted extensive experiments to compare the effectiveness of these four inputs in the gesture recognition, respectively. Then we proposed a semi-supervised leaning framework by utilizing few labeled data in the source domain and large amounts of unlabeled data in the target domain. Specially, we combine the ∏-model and some specific data augmentation tricks on the mmWave signal to realize the domain-independent gesture recognition. Extensive experiments on a public mmWave gesture dataset demonstrate the superior effectiveness of the proposed system.

Swimming Movement Guidance Training System Based on OpenCL Technology

In order to improve the physical quality and even ability of swimmers, this study built OpenCL technology for the swimming movement design system of swimmers, so as to design a series of movements. Combined with the capture system, this paper captures and processes the athletes’ movements, obtains the key position information, and calculates the accurate data. It can provide effective guidance for accurately positioning athletes’ swimming movements, enable users to have a clearer understanding of their own movements, improve users’ movements, and enhance the system operation effect. According to the technology and application requirements of virtual swimming training simulator, the overall design scheme of the device is determined. From the point of view of ensuring the comfort and safety of simulated virtual swimming, the design, installation, and strength verification of the supporting structure are carried out. Hardware design and selection shall be carried out according to the requirements of head position detection and virtual environment presentation, including the design of positioning ball set, camera selection, and VR head display selection. The software functions of head position detection program and virtual scene program are designed according to the requirements of natural human-computer interaction. The designed swimming guidance training system has complete functions and stable operation and can be used as an auxiliary tool for swimming training.

Review on Human Computer Interaction

In the field of artificial intelligence, human computer interaction(HCI) technology and its related intelligent robot technologies are essential and interesting contents of research. From the perspective of software algorithm and hardware system, we try to build a natural HCI environment. Human Computer Interaction(HCI) has been challenged in recent years because of advanced technology requiring adoption of new applications and investigations of connection with other disciplines ,to enhance its theoretical knowledge.

Design and Implementation of Virtual Roaming Control Method Based on Pen Tablet

A natural game interaction method can bring a good game experience to users, but pen-type interaction, which is one of the natural human-computer interaction methods, rarely appears in games, especially three-dimensional games. The main reason is that most pen-type interactive devices are developed for two-dimensional applications such as writing and painting. Whether the pen-based interaction can be extended from two-dimensional to three-dimensional is the key to whether the pen-based interaction can become one of the leading interactive methods of the game. The pen tablet is one of the most widely used pen-type interactive devices, and a virtual tour is an essential part of the game. Therefore, this paper takes this as a breakthrough and proposes a virtual roaming control method based on the pen tablet. First, we analyze the performance characteristics of the pen tablet and get the data information available for the pen tablet. Then, we design various virtual roaming actions such as walking, running, jumping, viewing angle rotation, view zooming, space-free displacement, acceleration, and deceleration and establish an association with the operation of the pen tablet. At the same time, the maximum likelihood function constraint was added to improve the error tolerance rate of viewing angle rotation. Finally, two games were developed to verify their effectiveness. The experimental results show that the pen tablet performs well in completing virtual roaming tasks, especially when users need to achieve similar writing and roaming tasks without switching devices back and forth. This maintains the user’s habit of writing with pen and paper and improves the interactive experience. The method is a new exploration of the application of pen-based interaction in games, which lays the foundation for future game development based on pen-based interaction.

Construction of Virtual Piano Performance System Based on Visual Gesture Recognition

In order for computers to understand people’s gestures through cameras and react accordingly, through in-depth research, gesture recognition technology in natural human-computer interaction is proposed. Combining natural human-computer interaction technology with music performance, using computer vision-based gestures, music is played in a virtual environment. Experiments show that the virtual piano has 14 piano keys. For the realization of piano performance, it is each piano key; once the piano key is greater than the set value, just call m_Wave.Load() to make a sound. According to the CWave class in the object-oriented MFC class library under VC++, create an object m_Wave of class CWave. Then, according to the m_Wave.Load() function of the CWave class, the connection of the sound is realized. It successfully solves the troubles of music lovers, enriches people’s spiritual life, and has certain practicability and scalability.

User-Defined Foot Gestures for Eyes-Free Interaction in Smart Shower Rooms

With the rapid development of natural human-computer interaction technologies, gesture-based interfaces have become popular. Although gesture interaction has received extensive attention from both academia and industry, most existing studies focus on hand gesture input, leaving foot-gesture-based interfaces underexplored, especially in scenarios where the user’s hands are occupied for other interaction tasks such as washing the hair in smart shower rooms. In such scenarios, users often have to perform interactive tasks (e.g., controlling water volume) with their eyes closed when water and shampoo liquid flow along with their head to eyes area. One possible way to address this problem is to use eyes-free (rather than eyes-engaged), foot-gesture-based interactive techniques that allow users to interact with the smart shower system without visual involvement. Through our online survey, 71.60% of the participants (58/81) have the requirements of using foot-gesture-based eyes-free interactions during showers. To this end, we conducted a three-phase study to explore foot-gesture-based interaction to achieve eyes-free interaction in smart shower rooms. We first derived a set of user-defined foot gestures for eyes-free interaction in smart shower rooms. Then, we proposed a taxonomy for foot gesture interaction. Our findings indicated that end-users preferred single-foot (76.1%), atomic (73.3%), deictic (65.0%), and dynamic (76.1%) foot gestures, which markedly differs from the results reported by previous studies on user-defined hand gestures. In addition, most of the user-defined dynamic foot gestures involve atomic movements perpendicular to the ground (40.1%) or parallel to the ground (27.7%). We finally distilled a set of concrete guidelines for foot gesture interfaces based on observing end-users’ mental model and behaviors when interacting with foot gestures. Our research can inform the design and development of foot-gesture-based interaction techniques for applications such as smart homes, intelligent vehicles, VR games, and accessibility design.

A Massage Area Positioning Algorithm for Intelligent Massage System.

A growing number of studies have been conducted over the past few years on the positioning of daily massage robots. However, most methods used for research have low interactivity, and a systematic method should be designed for accurate and intelligent positioning, thus compromising usability and user experience. In this study, a massage positioning algorithm with online learning capabilities is presented. The algorithm has the following main innovations: (1) autonomous massage localization can be achieved by gaining insights into natural human-machine interaction behavior and (2) online learning of user massage habits can be achieved by integrating recursive Bayesian ideas. As revealed by the experimental results, combining natural human-computer interaction and online learning with massage positioning is capable of helping people get rid of positioning aids, reducing their psychological and cognitive load, and achieving a more desirable positioning effect. Furthermore, the results of the analysis of user evaluations further verify the effectiveness of the algorithm.

Scene Construction and Application of Panoramic Virtual Simulation in Interactive Dance Teaching Based on Artificial Intelligence Technology

In recent years, with the rapid development of human motion simulation technology and virtual simulation technology, natural human-computer interaction has become the main form of research in the computer industry. Due to the research capabilities and practical value of mobile video technology, it is widely used in advanced fields, such as video animation production, rehabilitation medicine, sports training, and game software, effectively realizing the connection between the three-dimensional world and the simulation world. However, the application of motion imaging technology in teaching activities is not perfect. This article is based on dance teaching, adopts a dance movement analysis method based on vector matching, and uses artificial intelligence technology to create a virtual panoramic scene. Experimental data show that the average recognition rate of gestures is 89.8%, which can meet the gesture switching interaction of working conditions in virtual simulation of equipment operation.

A method of multimodal machine sign language translation for natural human-computer interaction

This paper aims to investigate the possibility of robustness enhancement as applied to an automatic system for isolated signs and sign languages recognition, through the use of the most informative spatiotemporal visual features. The authors present a method for the automatic recognition of gestural information, based on an integrated neural network model, which analyses spatiotemporal visual features: 2D and 3D distances between the palm and the face; the area of the hand and the face intersection; hand configuration; the gender and the age of signers. A 3DResNet-18-based neural network model for hand configuration data extraction was elaborated. Deepface software platform neural network models were embedded in the method in order to extract gender and age-related data. The proposed method was tested on the data from the multimodal corpus of sign language elements TheRuSLan, with the accuracy of 91.14 %. The results of this investigation not only improve the accuracy and robustness of machine sign language translation, but also enhance the naturalness of human-machine interaction in general. Besides that, the results have application in various fields of social services, medicine, education and robotics, as well as different public service centers.

Financial Big Data Management and Control and Artificial Intelligence Analysis Method Based on Data Mining Technology

Driven by capital and Internet information (IT) technology, the operating scale and capital scale of modern industrial and commercial enterprises and various organizations have increased exponentially. At present, the manual-based financial work model has been unable to adapt to the changing speed of the modern business environment and the business rhythm of enterprises. All kinds of enterprises and organizations, especially large enterprises, urgently need to improve the operational efficiency of financial systems. By enhancing the integrity, timeliness, and synergy of financial information, it improves the comprehensiveness and ability of analyzing complex problems in financial analysis. It can cope with such rapid changes and help improve the financial management capabilities of enterprises. It provides more valuable decision-making guidance for business operations and reduces business risks. In recent years, the vigorous development of artificial intelligence technology has provided a feasible solution to meet the urgent needs of enterprises. Combining data mining, deep learning, image recognition, natural language processing, knowledge graph, human-computer interaction, intelligent decision-making, and other artificial intelligence technologies with IT technology to transform financial processes, it can significantly reduce the processing time of repetitive basic financial processes, reduce the dependence on manual accounting processing, and improve the work efficiency of the financial department. Through the autonomous analysis and decision-making of artificial intelligence, the intelligentization of financial management is realized, and more accurate and effective financial decision-making support is provided for enterprises. This paper studies the company’s intelligent financial reengineering process, so as to provide reference and reference for other enterprises to upgrade similar financial systems. The results of the analysis showed that at the level of α = 0.05 , there was a significant difference in the mean between the two populations. When the r value is in the range of -1 and 1, the linear relationship between the x and y variables is more obvious. This paper proposes decision-making suggestions and risk control early warning to the group decision-making body, or evaluates the financial impact of the group’s decision-making, and opens the road to financial intelligence.

Gesture controlled interaction using hand pose model

Direct use of the hand as an input device is a smart method for providing natural human-computer interaction (HCI). Hand pose estimation is an attractive topic for research in recent years. It has been widely used in virtual reality. The domain of computer vision-based human hand three-dimensional shape and hand pose estimation has fascinated momentous attention recently due to its key role in various applications, such as natural human-computer interactions. Hand pose estimation is difficult due to some challenges. First, we need to detect the human hand which is very changeable. Second, the high degree of freedom leads to difficulties in pose estimation. In this paper, we aim to build a hand pose estimation system that can correctly detect a human hand and estimate its pose which can be useful in the areas like industrial automation, sign language recognition etc. We integrated a hand pose model with a game named U3D with 6 different gestures. As the object detection methods perform poorly in the palm detection tasks we used 21 hand joints and increased accuracy to approximately 95.63%. We used the Blender 3D computer graphics software toolset for creating animation, visual effects, motion graphics, and interactive unity 3D games.

Survey on Vision Based Hand Gesture Interface for Controlling Multimedia Player

As computers become additional pervasive in society, facilitating natural human–computer interaction (HCI) can have a positive impact on their use. Hence, there has been growing interest within the development of recent approaches and technologies for bridging the human-computer barrier. the last word aim is to bring HCI to a regime wherever interactions with computers are going to be as natural as associate degree interaction between humans, and to the current finish, incorporating gestures in HCI is a crucial analysis space. Gestures have long been thought-about as associate degree interaction technique that may doubtless deliver additional natural, creative, and intuitive strategies for human activity with our computers. Hand gesture recognition is one amongll|one amongst|one in every of} the systems that may notice the gesture of the hand in a period of time video. The gesture of hand is classed inside a definite space of interest. during this study, planning hand gesture recognition is one among the difficult jobs that involves 2 major issues. foremost is that the detection of the hand. Another drawback is to form an indication that's appropriate to be used one hand at a time. This project concentrates on however a system might notice, acknowledge and interpret hand gesture recognition through computer vision with the difficult factors that variability within the create, orientation, location, and scale. To perform well for developing this project, differing kinds of gestures like numbers and sign languages got to be created during this system. The image taken from the period of time video is analyzed via Haar-cascade Classifier to notice the gesture of hand before the image process is finished or in different words to notice the looks of hand in a very frame. during this project, the detection of hand are going to be done mistreatment the theories of Region of Interest (ROI) via Python programming. the reason of the results are going to be targeted on the simulation half since the distinction for the hardware implementation is that the ASCII text file to scan the period of time input video. the event of hand gesture recognition mistreatment Python, OpenCV, and YOLO V3 will be enforced by applying the theories of hand segmentation and also the hand detection system that uses the Haar-cascade classifier.

MTransSee

Gesture recognition using millimeter-wave radios facilitates natural human-computer interactions, but existing works require a consistent environment, i.e., the neural networks for recognition are trained and tested for the same users and at some fixed positions. In this case, their performance will decrease rapidly when they enter into a new environment. To make the model applicable in different environments, a straightforward approach is to collect and re-train the model for the gesture samples on every possible position upon each new user. However, it may ask the users to spend unacceptable time to accomplish such adaptation, which makes it difficult to be widely used in practice. In this paper, we first collect an abundant mmWave gesture dataset containing 59,280 samples as a benchmark to investigate the impact of the environment changes quantitatively. Then we propose a novel transfer-learning approach called mTransSee, which can serve the gestures in practice using pre-learned experience by least adaptation, i.e., retraining using only 8 samples per gesture for the same accuracy. mTransSee reduces dozens of workloads for the environment adaptation. We implement mTransSee on a commodity mmWave sensor and make a user study to compare the advance of mTransSee over the state-of-the-art solution in terms of user experience during adaptation.

Cross-Day EEG-Based Emotion Recognition Using Transfer Component Analysis

EEG-based emotion recognition can help achieve more natural human-computer interaction, but the temporal non-stationarity of EEG signals affects the robustness of EEG-based emotion recognition models. Most existing studies use the emotional EEG data collected in the same trial to train and test models, once this kind of model is applied to the data collected at different times of the same subject, its recognition accuracy will decrease significantly. To address the problem of EEG-based cross-day emotion recognition, this paper has constructed a database of emotional EEG signals collected over six days for each subject using the Chinese Affective Video System and self-built video library stimuli materials, and the database is the largest number of days collected for a single subject so far. To study the neural patterns of emotions based on EEG signals cross-day, the brain topography has been analyzed in this paper, which show there is a stable neural pattern of emotions cross-day. Then, Transfer Component Analysis (TCA) algorithm is used to adaptively determine the optimal dimensionality of the TCA transformation and match domains of the best correlated motion features in multiple time domains by using EEG signals from different time (days). The experimental results show that the TCA-based domain adaptation strategy can effectively improve the accuracy of cross-day emotion recognition by 3.55% and 2.34%, respectively, in the classification of joy-sadness and joy-anger emotions. The emotion recognition model and brain topography in this paper, verify that the database can provide a reliable data basis for emotion recognition across different time domains. This EEG database will be open to more researchers to promote the practical application of emotion recognition.

Review of Innovative Immersive Technologies for Healthcare Applications

ABSTRACTImmersive technologies, including virtual reality (VR), augmented reality (AR), and mixed reality (MR), can connect people using enhanced data visualizations to better involve stakeholders as integral members of the process. Immersive technologies have started to change the research on multidimensional genomic data analysis for disease diagnostics and treatments. Immersive technologies are highlighted in some research for health and clinical needs, especially for precision medicine innovation. The use of immersive technology for genomic data analysis has recently received attention from the research community. Genomic data analytics research seeks to integrate immersive technologies to build more natural human-computer interactions that allow better perception engagements. Immersive technologies, especially VR, help humans perceive the digital world as real and give learning output with lower performance errors and higher accuracy. However, there are limited reviews about immersive technologies used in healthcare and genomic data analysis with specific digital health applications. This paper contributes a comprehensive review of using immersive technologies for digital health applications, including patient-centric applications, medical domain education, and data analysis, especially genomic data visual analytics. We highlight the evolution of a visual analysis using VR as a case study for how immersive technologies step, can by step, move into the genomic data analysis domain. The discussion and conclusion summarize the current immersive technology applications' usability, innovation, and future work in the healthcare domain, and digital health data visual analytics.

Robust Hand Gesture Recognition Based on RGB-D Data for Natural Human–Computer Interaction

To naturally interact with virtual environment by hand gesture, this paper presents a robust RGB-D data based recognition method of static and dynamic hand gesture. Firstly, for static hand gesture recognition, starting from the hand gesture contour extraction, the palm center is identified by Distance Transform (DT) algorithm. The fingertips are localized by employing the K-Curvature-Convex Defects Detection algorithm (K-CCD). On the basis, the distances of the pixels on hand gesture contour to palm center and the angle between fingertips are considered as the auxiliary features to construct a multimodal feature vector, and then recognition algorithm is presented to robustly recognize the static hand gestures. Secondly, combining Euclidean distance between hand joints and shoulder center joint with the modulus ratios of skeleton features, this paper generates a unifying feature descriptor for each dynamic hand gesture and proposes an improved dynamic time warping (IDTW) algorithm to obtain recognition results of dynamic hand gestures. Finally, we conduct extensive experiments to test and verify the static and dynamic hand gesture recognition algorithm and realize a low-cost real-time application of natural interaction with virtual environment by hand gestures.

Natural multimodal interaction in immersive flow visualization

In the immersive flow visualization based on virtual reality, how to meet the needs of complex professional flow visualization analysis by natural human–computer interaction is a pressing problem. In order to achieve the natural and efficient human–computer interaction, we analyze the interaction requirements of flow visualization and study the characteristics of four human–computer interaction channels: hand, head, eye and voice. We give out some multimodal interaction design suggestions and then propose three multimodal interaction methods: head & hand, head & hand & eye and head & hand & eye & voice. The freedom of gestures, the stability of the head, the convenience of eyes and the rapid retrieval of voices are used to improve the accuracy and efficiency of interaction. The interaction load is balanced by multimodal interaction to reduce fatigue. The evaluation shows that our multimodal interaction has higher accuracy, faster time efficiency and much lower fatigue than the traditional joystick interaction.

Dynamic Gesture Recognition Using Surface EMG Signals Based on Multi-Stream Residual Network.

Gesture recognition technology is widely used in the flexible and precise control of manipulators in the assisted medical field. Our MResLSTM algorithm can effectively perform dynamic gesture recognition. The result of surface EMG signal decoding is applied to the controller, which can improve the fluency of artificial hand control. Much current gesture recognition research using sEMG has focused on static gestures. In addition, the accuracy of recognition depends on the extraction and selection of features. However, Static gesture research cannot meet the requirements of natural human-computer interaction and dexterous control of manipulators. Therefore, a multi-stream residual network (MResLSTM) is proposed for dynamic hand movement recognition. This study aims to improve the accuracy and stability of dynamic gesture recognition. Simultaneously, it can also advance the research on the smooth control of the Manipulator. We combine the residual model and the convolutional short-term memory model into a unified framework. The architecture extracts spatiotemporal features from two aspects: global and deep, and combines feature fusion to retain essential information. The strategy of pointwise group convolution and channel shuffle is used to reduce the number of network calculations. A dataset is constructed containing six dynamic gestures for model training. The experimental results show that on the same recognition model, the gesture recognition effect of fusion of sEMG signal and acceleration signal is better than that of only using sEMG signal. The proposed approach obtains competitive performance on our dataset with the recognition accuracies of 93.52%, achieving state-of-the-art performance with 89.65% precision on the Ninapro DB1 dataset. Our bionic calculation method is applied to the controller, which can realize the continuity of human-computer interaction and the flexibility of manipulator control.

Development of the Interactive Rehabilitation Game System for Children with Autism Based on Game Psychology

Autism is a widespread neurodevelopmental disorder. The cause of autism is still unclear. Once it occurs, it will be accompanied for life. Intervention treatment is currently recognized as an effective method. With the rise of somatosensory technology, Kinect has gradually been applied to the field of autism rehabilitation with its natural human-computer interaction and game rehabilitation psychology. Because there is a great demand for autism rehabilitation training resources in China and the existing somatosensory rehabilitation equipment has some shortcomings, this study designed a set of somatosensory rehabilitation system for autistic children based on Kinect and carried out its rehabilitation effect with experimental demonstration. The specific research work mainly includes the following contents: first, aiming at the rehabilitation needs of children with autism, this article combines mainstream intervention treatment methods, using C# programming language and Windows Presentation Foundation (WPF) development technology to develop Kinect-based somatosensory game modules. The somatosensory rehabilitation game involves the training of the language communication, behavioral movement, daily life, sensory integration, and other comprehensive abilities of children with autism, which improves the shortcomings of the previous rehabilitation training in the scope of intervention. A variety of game types and diversified game modes are designed to meet the individual needs of rehabilitation training for children with autism. Second, experiments were carried out to verify the rehabilitation effect of the system. Using the Childhood Autism Rating Scale (CARS) and Pediatric Evaluation of Disability Inventory (PEDI) scale as the rehabilitation effect evaluation tools of this system, the experimental subjects were divided into the experimental group and control group, and different comparative experiments were conducted, respectively. Through the analysis of the scoring data of the CARS, it is found that the comprehensive ability of the experimental group of children receiving somatosensory rehabilitation training is significantly improved. Through the analysis of the scoring data of the PEDI scale, it is found that the children’s language communication, behavioral activities, and daily life abilities of the experimental group have been significantly enhanced. This article focuses on the design of the somatosensory game module of the autistic somatosensory rehabilitation system and the analysis of the system’s rehabilitation effect. It satisfies the training and rehabilitation needs of children with autism and has strong practical significance.