Natural Human-computer Interaction Research Articles

EEG Dataset for the Recognition of Different Emotions Induced in Voice-User Interaction

Electroencephalography (EEG)-based open-access datasets are available for emotion recognition studies, where external auditory/visual stimuli are used to artificially evoke pre-defined emotions. In this study, we provide a novel EEG dataset containing the emotional information induced during a realistic human-computer interaction (HCI) using a voice user interface system that mimics natural human-to-human communication. To validate our dataset via neurophysiological investigation and binary emotion classification, we applied a series of signal processing and machine learning methods to the EEG data. The maximum classification accuracy ranged from 43.3% to 90.8% over 38 subjects and classification features could be interpreted neurophysiologically. Our EEG data could be used to develop a reliable HCI system because they were acquired in a natural HCI environment. In addition, auxiliary physiological data measured simultaneously with the EEG data also showed plausible results, i.e., electrocardiogram, photoplethysmogram, galvanic skin response, and facial images, which could be utilized for automatic emotion discrimination independently from, as well as together with the EEG data via the fusion of multi-modal physiological datasets.

Research on Mixed Reality Hand Interaction Technology in Flight Simulation Teaching

Abstract The integration of mixed reality simulation training in institutional education and teaching processes can yield significant quality benefits. Non-wearable interactive methods represent a more natural form of hand interaction within flight cockpits, covering various emerging fields such as mixed reality, computer vision, and human-computer interaction. This paper is guided by a “human-centered” approach to natural human-computer interaction and proposes a virtual hand mixed reality interaction scheme based on hand natural feature points. This study introduces a hand detection, segmentation, and recognition algorithm based on skin colour and key hand feature points. Initially, the algorithm uses the Otsu adaptive threshold algorithm in the YCbCr space for skin colour detection to accommodate varying image brightness levels. Subsequently, it employs a hand Keypoint model for rapid hand detection within skin-like areas while excluding interference from other regions. During the process of hand tracking, optimization prediction is performed using the particle filter algorithm combined with the artificial fish swarm algorithm to achieve tracking accuracy within six pixels in both horizontal and vertical directions. Gesture recognition involves extracting Fourier descriptive contour features of the hand bone structure through identification of feature points. Furthermore, this study combines the optimization capabilities of the artificial fish swarm algorithm to enhance support vector machine model parameter optimization for improved recognition rates. Testing involved 800 processed images as part of a test set for gesture recognition; only five commonly used gestures within flight cockpits were recognized and classified with an accuracy rate reaching up to 98%. Finally, this research presents virtual representations of common operational gestures including natural hand states, control stick manipulation, and throttle handling positions, button activations, and rotational movements via head-mounted displays or screens. The proposed mixed reality-based hand gesture interaction system addresses challenges related to complex background interference during hand detection in cockpit environments as well as issues arising from lighting disturbances. Additionally it resolves problems associated with insufficient particles leading to tracking failures while significantly enhancing registration effects during tracking processes. Moreover it mitigates issues related to Fourier descriptive features being influenced by background variations or changes in pose along with limitations pertaining to expressing singular forms of gestures thereby improving overall classification accuracy.

Visual Target Interaction Modeling with Multichannel Data Fusion

Facing the demand for accurate, fast, and natural human–computer interaction in the multi-screen control environment, the traditional exchange method of frequent switching control of multi-screen through mouse, keyboard, and other manual methods has problems of low efficiency, weak flexibility, and poor experience. In this paper, we research multi-screen precision control technology based on eye movement to break through the difficulties of precise recognition of human vision under complex environments and poor stability of vision estimation under frequent switching control of multi-screen. This study explores the technology for controlling multiple screens with high precision by tracking eye movements. It overcomes the challenges associated with reliably discerning human visual focus in intricate settings and the instability of visual assessments during rapid transitions between multiple screens. The experimental results of 1143 eye-movement capture tasks are studied, including the accurate recognition technology of human head posture and pupil gaze direction under complex backgrounds, illumination, and different visual field environments, and the theoretical modeling method of multi-screen precise target interaction based on eye-movement is explored. The expression data will be processed using convolutional neural networks, and the dataset will be trained through Python programming and tested on test samples collected by simulated flight crews in the laboratory with an accuracy rate of more than 85%.

Multimedia Human-Computer Interaction Technology in Preschool Education

With the increasing of computing and network technology, there has been a rise in children's educational and entertainment software. However, these software applications depend on mouse and keyboard input, which may be tedious and challenging for children. Moreover, these applications fail to capture the interaction between users and computers. This study proposes design principles for children's human-computer interaction based on children's cognitive, psychological, and physiological attributes, and outlines a framework accordingly. The framework integrates natural human-computer interaction technologies, such as physical operation interface, gesture interaction, low-level semantics-based human-computer interaction, and high-level semantics-based human-computer interaction, with the aim of facilitating seamless and intuitive interaction between children and computers. Finally, the results demonstrating the feasibility and effectiveness of the proposed human-computer interaction technology as a child-centered virtual reality application tool.

Desktop Voice Assistant

Abstract: Desktop voice assistants have become a popular interface for human-computer interaction because they allow users to speak with their computers in natural language. However, modern desktop voice assistants sometimes struggle to understand and respond to complex queries and context. Privacy concerns concerning the collection and use of user data also remain significant barriers. In order to enhance desktop voice assistants, we present a novel personalization and multimodal interaction technique in this research. By combining text, voice, and visual inputs, our approach improves the accuracy and relevance of the responses. By combining several modalities, our system is able to better understand context and user intent, leading to more meaningful interactions. Furthermore, we propose a user-centric personalization strategy whereby the voice assistant progressively learns each user's preferences and usage patterns. Because it depends less on centralized data collection and processing, this customized approach improves user experience while simultaneously addressing privacy concerns. Through a series of experiments and user surveys, we show how effective our approach is in improving the overall performance and user experience of desktop voice assistants. Our results show that the combination of multimodal interaction with personalization produces more accurate and contextually relevant responses, improving the user experience. All things considered, our research advances desktop voice assistant technology by addressing significant interface and personalization issues. Our approach has the potential to significantly improve desktop voice assistant usability and user acceptance, which could result in considerably more intuitive and natural human-computer interaction.

DigCode—A generic mid-air gesture coding method on human-computer interaction

With high flexibility and rich semantic expressiveness, mid-air gesture interaction is an important part of natural human-computer interaction (HCI) and has broad application prospects. However, there is no unified representation frame for designing, recording, investigating and comparing HCI mid-air gestures. Therefore, this paper proposes an interpretable coding method, DigCode, for HCI mid-air gestures. DigCode converts the unstructured continuous actions into structured discrete string encoding. From the perspective of human cognition and expression, the research employed psychophysical methods to divide gesture actions into discrete intervals, defined the coding rules of representation in letters and numbers, and developed automated programs to enable encoding and decoding by using gesture sensors. The coding method can cover the existing representations of HCI mid-air gestures by considering human understanding and computer recognition and can be applied to HCI mid-air gesture design and gesture library construction.

Product Design for the Elderly Based on Human-Computer Interaction in the Era of Big Data

Since 21st century, social aging trend has become more and more serious. It needs accurately identify the nursing needs of elderly disabled people, who are characterized by inconvenient movement and unclear speech. How to solve these problems has become a key point in the field of elderly care and medical care. In response to this problem, this research has designed a human-computer interactive gesture recognition system for elderly nursing beds in the context of big data. The recognition rate of the fusion feature + support vector machine (SVM) classifier adopted in this study is higher than 90% for each category of gesture. On the test set, this method has an average recognition rate of 96.35%, which is much higher than that of single feature + SVM classifier. While other methods’ recognition rate is lower than 90%. The recognition rate of tag c (nursing bed posture turning left) with obvious gesture feature information is as high as 99.28%, and that of tag h (nursing bed posture bedpan lowering) with weak gesture feature is 93.65%. The human-computer interaction system has well realized the recognition intention of user’s dynamic and static gestures, achieved the goal set by the research, and the interaction form is natural and reliable. In the later research, we can further realize a more comprehensive, accurate and natural human-computer interaction product design through the multi-channel joint decision-making scheme to meet the needs of the elderly.

A review on personal calibration issues for video-oculographic-based gaze tracking.

Personal calibration is a process of obtaining personal gaze-related information by focusing on some calibration benchmarks when the user initially uses a gaze tracking system. It not only provides conditions for gaze estimation, but also improves gaze tracking performance. Existing eye-tracking products often require users to conduct explicit personal calibration first, thereby tracking and interacting based on their gaze. This calibration mode has certain limitations, and there is still a significant gap between theoretical personal calibration methods and their practicality. Therefore, this paper reviews the issues of personal calibration for video-oculographic-based gaze tracking. The personal calibration information in typical gaze tracking methods is first summarized, and then some main settings in existing personal calibration processes are analyzed. Several personal calibration modes are discussed and compared subsequently. The performance of typical personal calibration methods for 2D and 3D gaze tracking is quantitatively compared through simulation experiments, highlighting the characteristics of different personal calibration settings. On this basis, we discuss several key issues in designing personal calibration. To the best of our knowledge, this is the first review on personal calibration issues for video-oculographic-based gaze tracking. It aims to provide a comprehensive overview of the research status of personal calibration, explore its main directions for further studies, and provide guidance for seeking personal calibration modes that conform to natural human-computer interaction and promoting the widespread application of eye-movement interaction.

Inferring in-air gestures in complex indoor environment with less supervision

People have high demands for comfort and technology in indoor environments. Gestures, as a natural and friendly human computer interaction (HCI) method, have received widespread attention and have been the subject of many research studies. Traditional approaches are based on wearable devices and cameras, which can be cumbersome to operate and infringe upon users’ privacy. Millimeter-wave (mmWave) radar avoids these problems by detecting gestures in a noninvasive manner. However, it encounters practical challenges in complex indoor environments, such as dynamic disturbance from surroundings, variable usage conditions and diverse gesture patterns, which conventionally require considerable manual effort to address. In this paper, we attempt to minimize human supervision and propose a noninvasive gesture recognition method named RaGe that involves a commercial mmWave indoor radar. First, a parameter optimization framework considering gesture prior constraints is proposed for radar configuration, which functions to weaken the disturbance from surroundings. Second, we alleviate data shortages in variable usage conditions and achieve low-cost data augmentation by applying affine transformations. Third, we combine deformable convolution operations with an unsupervised attention mechanism, thus exploring the intrinsic features involved in diverse gesture patterns. Experimental results show that RaGe is able to recognize 7 gestures with 99.3% accuracy and less human supervision, surpassing the state-of-the-art methods in comparative experiments.

Hand gesture recognition in natural human-computer interaction

This paper introduces the definition of Gesture recognition. First the article gives a precise definition of gesture recognition and explains the difference between gestures and postures, then it reveals technical difficulties of Gesture recognition. These technical difficulties include four aspects. After analyzing the technology and methods of Gesture recognition and the technical difficulties, it concretely expounds gesture recognition process based on data gloves, which is widely studied before and it also introduces the computer vision based research achievement which is currently becoming a research hotspot in this field. Then it also takes Kinect and HoloLens as examples to introduce specific practical cases of gesture recognition in wearable devices. Also it outlines the application of gesture recognition technology in human-computer interaction which include but not limited to smart terminal, Game control, Robot Control, Clinical and Health, Smart home, Sign Language Recognition, Vehicle system, Interactive entertainment. Finally it reach the conclusion that the biggest challenge researchers meet is to build a powerful framework to overcome the common problems with fewer constraints to provide reliable results. And sometimes researchers need to combine multiple methods for different complex environments.

Gesture-Based Control of Multimedia Player Using Python and OpenCV

The increasing significance of computers in our daily lives, coupled with the rise of ubiquitous computing, has necessitated effective human-computer interaction. Hand gesture recognition systems have emerged as a real-time video-based solution for detecting and interpreting hand gestures, offering intelligent and natural human-computer interaction (HCI) methods. This project focuses on leveraging human hands as input devices for computer operation. Developed using Python and the OpenCV library, the program utilizes a computer webcam to capture and analyze hand shapes and patterns. The program provides real-time feedback by displaying recognized hand gestures on the live video stream. The ultimate outcome of this project is an application that enhances user experiences in contactless systems. The project recognizes and detects human hand motions using the Python computer language through a process flow that includes background subtraction, hand ROI segmentation, contour detection, and finger recognition. Techniques for processing images are used, including hand gesture detection, pattern recognition, thresholding, and contour detection. The processing of incoming photos and the creation of related keystrokes are made possible by OpenCV, a rich set of image processing tools

Future Interdisciplinary Combination of AI Technologies and Psychology

The paper aims to address several interdisciplinary and multidisciplinary research issues at the confluence of psychology with the field of artificial intelligence technologies, namely how the integration of big data analytics, advancements in human-computer interaction (HCI), and innovations in brain-computer interfaces (BCIs), can transform psychological research and practice. The paper explores the historical foundations and contemporary developments in the interdisciplinary integration of AI technologies with psychological research and practice. Beginning with early computational models of cognition, the discussion highlights the evolution of AI applications in psychological analysis, including machine learning, natural language processing (NLP), HCI and BCIs. These technologies have not only enhanced the scalability and personalization of mental health care but have also introduced new methods for real-time feedback in therapeutic settings. This paper provides a wide examination of how these interdisciplinary efforts can complement and advance both fields, encouragement mutual development and innovation.Article’s history: Received 1st of September, 2023; Received in revised form 20th of September, 2023; Accepted 21st of October, 2023; Published as article in Volume I, Issue 1, 2023.Copyright© 2023. The Author(s). This article is distributed under the terms of the license CC-BY 4.0., which permits any further distribution in any medium, provided the original work is properly cited.Keywords: artificial intelligence, big data, brain-computer interface, human computer interaction.

A Deep Graph Network with Multiple Similarity for User Clustering in Human–Computer Interaction

User counterparts, such as user attributes in social networks or user interests, are the keys to more natural Human–Computer Interaction (HCI) . In addition, users’ attributes and social structures help us understand the complex interactions in HCI. Most previous studies have been based on supervised learning to improve the performance of HCI. However, in the real world, owing to signal malfunctions in user devices, large amounts of abnormal information, unlabeled data, and unsupervised approaches (e.g., the clustering method) based on mining user attributes are particularly crucial. This paper focuses on improving the clustering performance of users’ attributes in HCI and proposes a deep graph embedding network with feature and structure similarity (called DGENFS ) to cluster users’ attributes in HCI applications based on feature and structure similarity. The DGENFS model consists of a Feature Graph Autoencoder (FGA) module, a Structure Graph Attention Network (SGAT) module, and a Dual Self-supervision (DSS) module. First, we design an attributed graph clustering method to divide users into clusters by making full use of their attributes. To take full advantage of the information of human feature space, a k-neighbor graph is generated as a feature graph based on the similarity between human features. Then, the FGA and SGAT modules are utilized to extract the representations of human features and topological space, respectively. Next, an attention mechanism is further developed to learn the importance weights of different representations to effectively integrate human features and social structures. Finally, to learn cluster-friendly features, the DSS module unifies and integrates the features learned from the FGA and SGAT modules. DSS explores the high-confidence cluster assignment as a soft label to guide the optimization of the entire network. Extensive experiments are conducted on five real-world data sets on user attribute clustering. The experimental results demonstrate that the proposed DGENFS model achieves the most advanced performance compared with nine competitive baselines.

Camera-based interactive wall display using hand gesture recognition

the recognition of Hand gestures has become a critical point as it is widely used in everyday applications. the challenge in this is to improve the recognition effect and develop a fast recognition method. Glove and led-based methods involve external devices in detecting and interpreting hand gestures, making human-computer interaction less natural. So, different approaches have been used previously that use purely hand gestures in many systems based on human-computer interaction. This system provides a more natural human-computer interaction; it must be made efficient processing speed of classifying the test data (images) from among the training data (database stored for gestures recognition). This speed makes gesture recognition more effective and reliable to use as compared to previously proposed methods. In this research paper, a proposed system based on a camera-based interactive wall display using bare hand gestures with efficient processing speed for controlling the speed of the mouse and other functions. This system has three modules: one uses Genetic Algorithm and Otsu thresholding to identify the query images as the right or wrong gesture and perform the correct action in case of the proper motion, another module controls functions outside of PowerPoint files or Word documents, e.g., to open folders and go through drives, and the third module uses the convexity hull method for finding the number of fingers open in the user's gesture and operates accordingly.

Survey on Vision Based Hand Gesture Interface for Controlling Multimedia Player

As computers become additional pervasive in society, facilitating natural human–computer interaction (HCI) can have a positive impact on their use. Hence, there has been growing interest within the development of recent approaches and technologies for bridging the human-computer barrier. the last word aim is to bring HCI to a regime wherever interactions with computers are going to be as natural as associate degree interaction between humans, and to the current finish, incorporating gestures in HCI is a crucial analysis space. Gestures have long been thought-about as associate degree interaction technique that may doubtless deliver additional natural, creative, and intuitive strategies for human activity with our computers. Hand gesture recognition is one amongll|one amongst|one in every of} the systemsthat may notice the gesture of the hand in a period of time video. The gesture of hand is classed inside a definite space of interest. during this study, planning hand gesture recognition is one among the difficult jobs that involves 2 major issues. foremost is that the detection of the hand. Another drawback is to form an indication that's appropriate to be used one hand at a time. This project concentrates on however a system might notice, acknowledge and interpret hand gesture recognition through computer vision with the difficult factors that variability within the create, orientation, location, and scale. To perform well for developing this project, differing kinds of gestures like numbers and sign languages got to be created during this system. The image taken from the period of time video is analyzed via Haar-cascade Classifier to notice the gesture of hand before the image process is finished or in different words to notice the looks of hand in a very frame. during this project, the detection of hand are going to be done mistreatment the theories of Region of Interest (ROI) via Python programming. the reason of the results are going to be targeted on the simulation half since the distinction for the hardware implementation is that the ASCII text file to scan the period of time input video. the event of hand gesture recognition mistreatment Python, OpenCV, and YOLO V3 will be enforced by applying the theories of hand segmentation and also the hand detection system that uses the Haar-cascade classifier

Werewolf-XL: A Database for Identifying Spontaneous Affect in Large Competitive Group Interactions

Affective computing and natural human-computer interaction, which would be capable of interpreting and responding intelligently to the social cues of interaction in crowds, are more needed than ever as an individuals affective experience is often related to others in group activities. To develop the next-generation intelligent interactive systems, we require numerous human facial expressions with accurate annotations. However, existing databases usually consider nonspontaneous human behavior (posed or induced), individual or dyadic setting, and a single type of emotion annotation. To address this need, we created the Werewolf-XL database, which contains a total of 890 minutes of spontaneous audio-visual recordings of 129 subjects in a group interaction of nine individuals playing a conversational role-playing game called Werewolf. We provide 131,688 individual utterance-level video clips with internal self-assessment of 18 non-prototypical emotional categories and external assessment of pleasure, arousal, and dominance, including 14,632 speakers' samples and the rest of listeners' samples. Besides, the results of the annotation agreement analysis show fair reliability and validity. Role information and outcomes of the game are also recorded. Furthermore, we provided extensive benchmarks of unimodal and multimodal emotional recognition results. The database is made publicly available.

A fitting algorithm based on multi-touch gesture for rapid generation of railway line

Human-computer interaction (HCI) technology plays a critically essential role in the computer-aided design of railway line locations. However, the traditional interactive design with a mouse+keyboard cannot well meet the rapid generation requirements of the railway line during scheme discussion. This research presents a fitting algorithm for the rapid generation of railway lines by using a multi-touch gesture algorithm. The fitting method from free hand-drawing lines to satisfied railway lines is proposed. Then the interactive operation hand gestures are defined and implemented into the railway line location design. The hand-drawing lines generated by defined gestures are automatically fitted with the target horizontal line by using the inflection detection algorithm based on Euclidean Distance (ED). Meanwhile, the vertical line can be fitted by a similar algorithm with an extreme point-to-point (EPP) and chord-to-point distance accumulation (CPDA). Moreover, a real-world example verification is carried out. The multi-touch gesture algorithm is applied for the automatic fitting of the railway line. Compared with the traditional interactive methods, the consumption time of railway line generation by using the multi-touch interactive mode is decreased by about 15%. This research provides fundamental support for rapid scheme discussion of railway line generation based on natural HCI, which is well-matched with modern handheld devices, and the requirements of rapid selection as well as the quick comparison of railway line schemes in the early stage of design.

Selection of features and hidden Markov model parameters for English word recognition from Leap Motion air‐writing trajectories

AbstractAir‐writing recognition is relevant in areas such as natural human–computer interaction, augmented reality, and virtual reality. A trajectory is the most natural way to represent air writing. We analyze the recognition accuracy of words written in air considering five features, namely, writing direction, curvature, trajectory, orthocenter, and ellipsoid, as well as different parameters of a hidden Markov model classifier. Experiments were performed on two representative datasets, whose sample trajectories were collected using a Leap Motion Controller from a fingertip performing air writing. Dataset contains 840 English words from 21 classes, and dataset contains 1600 English words from 40 classes. A genetic algorithm was combined with a hidden Markov model classifier to obtain the best subset of features. Combination trajectory, orthocenter, writing direction, curvature provided the best feature set, achieving recognition accuracies on datasets and of 98.81% and 83.58%, respectively.

Evaluation of 6 DOF Robotic Arm Using Leap Motion Sensor

Due to their increased functionality, robotic arms provide a well-organized method for developing assistive devices. By 2035, statistics indicate that half of Germany’s population will be over the age of fifty, and every third person will be over the age of sixty. These aging societies face numerous obstacles when it comes to performing basic activities of daily living, or ""ADLs."" A growing body of research is focusing on Ambient Assisted Living, or ""AAL,"" as a novel approach to addressing the needs of elderly people. A critical objective of AAL is to improve the quality of life for the elderly and disabled and to assist them in maintaining an independent lifestyle. Robotics and technology-enabled environments will be critical in enabling elderly and physically disabled people to maintain a self-determined, independent lifestyle in their familiar surroundings. The purpose of this article is to propose the implementation of a novel intuitive and adaptive manipulation scheme by creating a human-machine communication interface between the Leap Motion controller and the 6-DOF Jaco robotic arm. An algorithm is developed to optimize the mapping between the user’s hand movement and the Jaco arm, as tracked by the Leap Motion controller. By constantly adapting to the user’s hand tremor or shake, the system should enable a more natural human-computer interaction and smooth manipulation of the robotic arm. The implementation would significantly improve people’s quality of life, particularly those with upper limb problems, by assisting them in performing several essential Activities of Daily Living ""ADLs."" The applications of this human-robot interaction will be discussed in relation to Ambient Assisted Living, with the introduction of several use case scenarios.

Magnetic Motion Tracking for Natural Human Computer Interaction: A Review

In the field of multimodal human–computer interaction (HCI), hand (and finger) motion tracking remains a critical challenge because it is the input means of visual reproduction and provides the necessary parameters for real-time modeling of the haptic model. Among motion tracking methods including optical tracking methods, inertial tracking methods, and magnetic motion tracking (MMT) methods, the MMT method has a great development prospect as a marker-based method because of its advantages of no line-of-sight problem, being natural-oriented, and high accuracy. In this article, we first discuss the performance and application scenarios under different configurations of electromagnet-based MMT (EMMT) and permanent-magnet-based MMT (PMMT) systems. And then, the MMT algorithms are reviewed, which can be divided into two broad categories: The model-based algorithms include linear algorithm, nonlinear optimization algorithm, and recursive Bayesian algorithm, and the offline-data-based algorithms include neural network algorithm and lookup table (LUT) algorithm. In addition, a state-of-the-art comparison of MMT algorithms is given. In the end, we have an insightful discussion and give the expected future outlook of MMT.