Human-computer Interaction Scenarios Research Articles

Identification of Object Roughness Using a Flexible Capacitive 3D Force Transducer Featuring an Interlocking Microstructure

A flexible capacitive three-dimensional (3D) force transducer with four bottom electrodes is proposed. A dielectric layer with ionic effects and pyramid-like microstructures is designed to improve the sensing performance of this sensor. The arrayed sensor exhibits remarkable capabilities in capturing and decoupling 3D forces, showcasing a high sensitivity, a wide response range (0–30[Formula: see text]kPa), rapid response time (62[Formula: see text]ms), excellent repeatability under continuous pressure, as well as static and dynamic responses. By assembling a robotic arm to grasp various objects, the sensor demonstrates exceptional sensing capabilities for object roughness, thereby highlighting its potential application as a touch recognition system in human–computer interaction scenarios.

Design and Optimization of Human-Computer Interaction System for Education Management Based on Artificial Intelligence

The continuous improvement and refinement of artificial intelligence (AI) technology has facilitated the broader application of human-computer interaction in the field of education management. The construction of an educational management human-computer interaction system based on AI technology can optimize and improve key parameters of educational management human-computer interaction scenarios, thereby creating a more comprehensive mobile learning (m-learning) application system. This paper is based on AI technology, analyzing gesture semantics and speech semantics, and combining fusion algorithms to construct an education management human interaction system. The performance changes of the system were compared with real experimental operations and the NOBOOK platform analysis. The results show that the education management human-computer interaction system constructed in this article can enhance the m-learning experience of participants. It ensures high recognition accuracy, leading to higher scores in all dimensions of indicator evaluation. Therefore, as one of the crucial forms of m-learning, the human-computer interaction system for education management based on AI can establish a foundation for the further enhancement and development of education management.

Incorporating Machine Learning Strategies to Smart Gloves Enabled by Dual-Network Hydrogels for Multitask Control and User Identification.

Smart gloves are often used in human-computer interaction scenarios due to their portability and ease of integration. However, their application in the field of information security has been less studied. Herein, we propose a smart glove using an iontronic capacitive sensor with significant pressure-sensing performance. Besides, an operator interface has been developed to match the smart glove, which is capable of multitasking integration of mouse movement, music playback, game control, and message typing in Internet chat rooms by capturing and encoding finger-tapping movements. In addition, by integrating machine learning, we can mine the characteristics of individual behavioral habits contained in the sensor signals and, based on this, achieve a deep binding of the user to the smart glove. The proposed smart glove can greatly facilitate people's lives, as well as explore a new strategy in research on the application of smart gloves in data security.

Emotional Experience during Human-Computer Interaction: A Survey

As human-computer interaction (HCI) technology becomes more and more integrated into our daily life, increasing attention has been drawn towards the interaction experience in addition to HCI efficiency. In the present study, we conducted a survey to explore context-specific emotional experience in HCI. Four hundred participants were recruited to report the frequency of their emotional experiences on 44 fine-grained emotion items in six representative HCI scenarios. Compared with six matched human-human interaction (HHI) scenarios used as control, the HCI scenarios were in general more frequently associated with negative emotions, and less frequently associated with positive emotions, especially when computer served as a tool for communication with other people. Furthermore, the 44 emotional experience items in HCI were summarized as five factors, representing low-arousal focused, positively engaged, emotionally empathetic, high-arousal negative and frustratingly confused. Our study presents a comprehensive overview of context-specific emotional experience in human-computer interactions and provides a framework for emotion evaluation in HCI applications.

Virtual Mouse – Hand Gesture Recognition System

The purpose of this research is to enhance the identification of hand gestures in a Human-Computer Interaction scenario, while also minimizing the computational time required and enhancing user comfort. To achieve this goal, the authors created a computer mouse control application that employs an algorithm and selected hand features, resulting in efficient performance. The integration of voice assistant with the recommended hand postures further improves user experience and facilitates system operation.

AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Persons

Automatically generating videos in which synthesized speech is synchronized with lip movements in a talking head has great potential in many human-computer interaction scenarios. In this paper, we present an automatic method to generate synchronized speech and talking-head videos on the basis of text and a single face image of an arbitrary person as input. In contrast to previous text-driven talking head generation methods, which can only synthesize the voice of a specific person, the proposed method is capable of synthesizing speech for any person. Specifically, the proposed method decomposes the generation of synchronized speech and talking head videos into two stages, i.e., a text-to-speech (TTS) stage and a speech-driven talking head generation stage. The proposed TTS module is a face-conditioned multi-speaker TTS model that gets the speaker identity information from face images instead of speech, which allows us to synthesize a personalized voice on the basis of the input face image. To generate the talking head videos from the face images, a facial landmark-based method that can predict both lip movements and head rotations is proposed. Extensive experiments demonstrate that the proposed method is able to generate synchronized speech and talking head videos for arbitrary persons, in which the timbre of the synthesized voice is in harmony with the input face, and the proposed landmark-based talking head method outperforms the state-of-the-art landmark-based method on generating natural talking head videos.

A Task Modeling Method of Intelligent Human–Computer Interaction in Aircraft Cockpits Based on Information Load Flow

Defining how tasks should be performed is an essential prerequisite for designing the next-generation aircraft cockpits with multimodal intelligent human–computer interaction (HCI) modes. To this end, a task modeling method based on information load flow is proposed in this article, namely weighted social network analysis (WSNA). In the WSNA method, the node relation complexity in a weighted social network is quantified based on entropy, and composite matrices are established to describe the association frequency and strength between linked nodes. The performance evaluation indicators are proposed to measure the quality of weighted social networks on node centrality, node strength, network density, and network robustness, which contributes to identify the overload nodes and edges. The transformation rules from traditional modes to intelligent modes are also created as guidelines for application. A case study set in an HCI scenario during the approach phase is given for illustration. The information load and overall performance are compared in these two task modes. The results verify the feasibility of WSNA and thereby provide a solution to planning the HCI tasks in next-generation aircraft cockpits.

Asynchronous H[formula omitted] filtering for semi-Markov jump T-S fuzzy systems within partial state delay and deception attack: Applied to aircraft-pilot state estimation

This article addresses an asynchronous filter design problem within partial state delay and deception attack for a class of discrete-time semi-Markov jump Takagi-Sugeno (T-S) systems which can describe estimation process for human pilot cognitive property. First, the emission probability, which relates to system mode and filtering mode, is introduced to consider the system process with asynchronously observation scheme. Second, state delay-independent condition and state delay-dependent condition designed within state delay boundary are proposed under the H∞ performance index γ. Third, the system output signal under deception attack is adopted as the information interference during the transmitted filter process. Through the Lyapunov-Krasovskii approach, stability of the proposed filtering scheme that is mode dependent can be guaranteed. Moreover, the filtering mode is also dynamically changed according to the prescribed asynchronous formula. A numerical example considered for aircraft-pilot state estimation is used to demonstrate the effectiveness and applicability of our proposed filtering method by obtaining the minimum γ. Experiments show that filter could play a better role in advancing signal estimation in aircraft-pilot state estimation scenario with properties like state delay and deception attack. Also, it contributes to modelling the fully human pilot cognitive decision-making in a human-computer interaction scenario.

New Application: A Hand Air Writing System Based on Radar Dual View Sequential Feature Fusion Idea

In recent years, non-contact human–computer interactions have aroused much attention. In this paper, we mainly propose a dual view observation system based on the frontal and side millimeter-wave radars (MWR) to collect echo data of the Air writing digits “0~9”, simultaneously. Additionally, we also propose a novel distance approximation method to make the trajectory reconstruction more efficient. To exploit these characteristics of spatial-temporal adjacency in handwriting digits, we propose a novel clustering algorithm, named the constrained density-based spatial clustering of application with noise (CDBSCAN), to remove background noise or clutter. Moreover, we also design a robust gesture segmentation method by using twice-difference and high–low thresholds. In our trials and comparisons, based on the trajectories formulated by echo data series of time–distance and time–velocity of dual views, we present a lightweight-based convolution neural network (CNN) to realize these digits recognition. Experiment results show that our system has a relatively high recognition accuracy, which would provide a feasible application for future human–computer interaction scenarios.

Learning End-User Customized Mid-Air Hand Gestures Using a Depth Image Sensor

Interacting with computer applications using actions that are designed by end users themselves instead of pre-defined ones has advantages such as better memorability in some Human-Computer Interaction (HCI) scenarios. In this paper we propose a method for allowing users to use self-defined mid-air hand gestures as commands for HCI after they provide a few training samples for each gesture in front of a depth image sensor. The gesture detection and recognition algorithm is mainly based on pattern matching using 3 separate sets of features, which carry both finger-action and hand-motion information. An experiment in which each subject designed their own set of 8 gestures, provided about 5 samples for each, and then used them to play a game is conducted all in one sitting. During the experiment a recognition rate of 66.7% is achieved with a false positive ratio of 22.2%. Further analyses on the collected dataset shows that a higher recognition rate of up to about 85% can be achieved if more wrong detections were allowed.

MsEmoTTS: Multi-Scale Emotion Transfer, Prediction, and Control for Emotional Speech Synthesis

Expressive synthetic speech is essential for many human-computer interaction and audio broadcast scenarios, and thus synthesizing expressive speech has attracted much attention in recent years. Previous methods performed the expressive speech synthesis either with explicit labels or with a fixed-length style embedding extracted from reference audio, both of which can only learn an average style and thus ignores the multi-scale nature of speech prosody. In this paper, we propose MsEmoTTS, a multi-scale emotional speech synthesis framework, to model the emotion from different levels. Specifically, the proposed method is a typical attention-based sequence-to-sequence model and with proposed three modules, including global-level emotion presenting module (GM), utterance-level emotion presenting module (UM), and local-level emotion presenting module (LM), to model the global emotion category, utterance-level emotion variation, and syllable-level emotion strength, respectively. In addition to modeling the emotion from different levels, the proposed method also allows us to synthesize emotional speech in different ways, i.e., transferring the emotion from reference audio, predicting the emotion from input text, and controlling the emotion strength manually. Extensive experiments conducted on a Chinese emotional speech corpus demonstrate that the proposed method outperforms the compared reference audio-based and text-based emotional speech synthesis methods on the emotion transfer speech synthesis and text-based emotion prediction speech synthesis respectively. Besides, the experiments also show that the proposed method can control the emotion expressions flexibly. Detailed analysis shows the effectiveness of each module and the good design of the proposed method.

FMCW Radar-Based Real-Time Hand Gesture Recognition System Capable of Out-of-Distribution Detection

Real-time hand gesture recognition plays a vital role in human-computer interaction (HCI). Recent radar-based hand gesture recognition methods have focused on achieving high classification accuracy using deep neural network (DNN)-based classifiers. However, the hand gesture recognition system should not only classify the gestures accurately but also detect out-of-distribution (OOD) samples to be used in real-world HCI scenarios with high reliability. Recognition systems without OOD detection capability misclassify unintended gestures in silence, especially in real-time scenarios. To tackle this problem, we propose a real-time hand gesture recognition system that can simultaneously classify hand gestures and detect OOD samples by using a Frequency Modulated ContinuousWave (FMCW) radar sensor. First, we design radar data processing technique and Transformer encoder-based classifier to achieve high classification accuracy. Second, the relative Mahalanobis distance (RMD)-based OOD detection method is adopted to increase the reliability of the proposed system. Finally, one in-distribution dataset and two OOD datasets are collected to verify the proposed system. The proposed system achieves a classification accuracy of 93.95%on the in-distribution dataset.We conduct the OOD detection experiments with two OOD datasets for which the proposed system reports AUROC values of 92.96% and 92.84%, respectively. Furthermore, the feasibility of the proposed system is certified through a real-time experimental demonstration.

A Cross-Domain Augmentation-Based AI Learning Framework for In-Network Gesture Recognition

This article studies the problem of RFID-based gesture recognition, which is practically important in various human-computer interaction scenarios, for example, smart homes, intelligent logistics, and smart cities. However, the existing solutions normally suffer from two major limitations: the model-driven methods are sensitive to specific environmental factors, and usually do not adapt well to a complex scenario that is full of multipath; the data-driven methods normally need the collection of massive RFID training data, and deploying the model in the remote cloud leads to long response delay. To overcome the above limitations, this article proposes a cross-domain augmentation-based AI learning (CAL) framework in the context of cloud-edge computing. In the CAL framework, we can simulate massive RFID phase profiles by converting the computer vision data that contains the gesture movement information, instead of costing lots of manpower to actually collect RFID training data. The simulated RFID phase profiles are used to train an AI model in the high-performance cloud. Note that since many sources of this kind of computer vision data are available online, we actually do not even need any manpower to collect training data. To achieve time-efficient recognition, knowledge distillation is applied to get a light and accurate model, which is deployed at the edge side. Thus, recognition response delay can be significantly reduced because the edge server where the AI model is actually deployed is much closer to users than the cloud server. We use commercial off-the-shelf RFID, Kinect, a high-performance server, and a laptop to implement the CAL framework. Extensive experiments are conducted to evaluate the performance of CAL. The results reveal that gesture recognition accuracy of CAL can reach nearly 90 percent without collection of any RFID training data.

Opportunistic Activity Recognition in IoT Sensor Ecosystems via Multimodal Transfer Learning

Recognizing human activities seamlessly and ubiquitously is now closer than ever given the myriad of sensors readily deployed on and around users. However, the training of recognition systems continues to be both time and resource-consuming, as datasets must be collected ad-hoc for each specific sensor setup a person may encounter in their daily life. This work presents an alternate approach based on transfer learning to opportunistically train new unseen or target sensor systems from existing or source sensor systems. The approach uses system identification techniques to learn a mapping function that automatically translates the signals from the source sensor domain to the target sensor domain, and vice versa. This can be done for sensor signals of the same or cross modality. Two transfer models are proposed to translate recognition systems based on either activity templates or activity models, depending on the characteristics of both source and target sensor systems. The proposed transfer methods are evaluated in a human–computer interaction scenario, where the transfer is performed in between wearable sensors placed at different body locations, and in between wearable sensors and an ambient depth camera sensor. Results show that a good transfer is possible with just a few seconds of data, irrespective of the direction of the transfer and for similar and cross sensor modalities.

Design and Development of Mathematical Models for Human-Computer Interaction

Human-Computer Interaction (HCI) is a key factor in determining how easily and effectively computers can be used. Recent years have seen a substantial increase in interest in the design and development of mathematical models for HCI due to their potential to improve user experience and boost system performance. In-depth analysis of the creation and development of mathematical models for HCI is presented in this research work, with an emphasis on their uses, difficulties, and potential future approaches. The many mathematical modelling approaches used in HCI are discussed in the study, including statistical models, machine learning techniques, and cognitive models. It looks at their benefits and drawbacks and emphasises the significance of using user-centered design principles all the way through the modelling process. The paper also looks at case studies that show the practical application and assessment of mathematical models in real-world HCI Scenario. Finally, it highlights new directions for study and emerging trends in the area of mathematical models for HCI.

The uulmMAC Database—A Multimodal Affective Corpus for Affective Computing in Human-Computer Interaction

In this paper, we present a multimodal dataset for affective computing research acquired in a human-computer interaction (HCI) setting. An experimental mobile and interactive scenario was designed and implemented based on a gamified generic paradigm for the induction of dialog-based HCI relevant emotional and cognitive load states. It consists of six experimental sequences, inducing Interest, Overload, Normal, Easy, Underload, and Frustration. Each sequence is followed by subjective feedbacks to validate the induction, a respiration baseline to level off the physiological reactions, and a summary of results. Further, prior to the experiment, three questionnaires related to emotion regulation (ERQ), emotional control (TEIQue-SF), and personality traits (TIPI) were collected from each subject to evaluate the stability of the induction paradigm. Based on this HCI scenario, the University of Ulm Multimodal Affective Corpus (uulmMAC), consisting of two homogenous samples of 60 participants and 100 recording sessions was generated. We recorded 16 sensor modalities including 4 × video, 3 × audio, and 7 × biophysiological, depth, and pose streams. Further, additional labels and annotations were also collected. After recording, all data were post-processed and checked for technical and signal quality, resulting in the final uulmMAC dataset of 57 subjects and 95 recording sessions. The evaluation of the reported subjective feedbacks shows significant differences between the sequences, well consistent with the induced states, and the analysis of the questionnaires shows stable results. In summary, our uulmMAC database is a valuable contribution for the field of affective computing and multimodal data analysis: Acquired in a mobile interactive scenario close to real HCI, it consists of a large number of subjects and allows transtemporal investigations. Validated via subjective feedbacks and checked for quality issues, it can be used for affective computing and machine learning applications.

Natural language understanding approaches based on joint task of intent detection and slot filling for IoT voice interaction

Internet of Things (IoT) based voice interaction system, as a new artificial intelligence application, provides a new human–computer interaction mode. The more intelligent and efficient communication approach poses greater challenges to the semantic understanding module in the system. Facing with the complex and diverse interactive scenarios in practical applications, the academia and the industry urgently need more powerful Natural Language Understanding (NLU) methods as support. Intent Detection and Slot Filling joint task, as one of the core sub-tasks in NLU, has been widely used in different human–computer interaction scenarios. In the current era of deep learning, the joint task of Intent Detection and Slot Filling has also changed from previous rule-based methods to deep learning-based methods. It is an important problem to explore how to realize the models of these tasks to be refined and targeted designed, and to make the Intent Detection task better serve the improvement of precision of Slot Filling task by connecting the before and after tasks. It has great significance for building a more humanized IoT voice interaction system. In this study, we designed two joint models to realize Intent Detection and Slot Filling joint task. For the Intent Detection type task, one is based on BiGRU-Att-CapsuleNet (hybrid-based model) and the other is based on the RCNN model. Both methods use the BiGRU-CRF model for the Slot Filling type task. The hybrid-based model can enhance the semantic capture capability of a single model. And by combining specialized models built independently for each task to achieve a complete joint task, it can be better to achieve optimal performance on each task. This study also carried out detailed comparative experiments of tasks and joint tasks on multiple datasets. Experiments show that the joint models have achieved competitive results in 7 typical datasets included in multiple scenarios in English and Chinese compared with other models.

Continuous Distant Measurement of the User's Heart Rate in Human-Computer Interaction Applications.

In real world scenarios, the task of estimating heart rate (HR) using video plethysmography (VPG) methods is difficult because many factors could contaminate the pulse signal (i.e., a subjects’ movement, illumination changes). This article presents the evaluation of a VPG system designed for continuous monitoring of the user’s heart rate during typical human-computer interaction scenarios. The impact of human activities while working at the computer (i.e., reading and writing text, playing a game) on the accuracy of HR VPG measurements was examined. Three commonly used signal extraction methods were evaluated: green (G), green-red difference (GRD), blind source separation (ICA). A new method based on an excess green (ExG) image representation was proposed. Three algorithms for estimating pulse rate were used: power spectral density (PSD), autoregressive modeling (AR) and time domain analysis (TIME). In summary, depending on the scenario being studied, different combinations of signal extraction methods and the pulse estimation algorithm ensure optimal heart rate detection results. The best results were obtained for the ICA method: average RMSE = 6.1 bpm (beats per minute). The proposed ExG signal representation outperforms other methods except ICA (RMSE = 11.2 bpm compared to 14.4 bpm for G and 13.0 bmp for GRD). ExG also is the best method in terms of proposed success rate metric (sRate).

Towards health monitoring using remote heart rate measurement using digital camera: A feasibility study

The paper presents a feasibility study for heart rate measurement using a digital camera to perform health monitoring. The feasibility study investigates the reliability of the state of the art heart rate measuring methods in realistic situations. Therefore, an experiment was designed and carried out on 45 subjects to investigate the effects caused by illumination, motion, skin tone, and distance variance. The experiment was conducted for two main scenarios; human-computer interaction scenario and health monitoring scenario. The human-computer scenario investigated the effects caused by illumination variance, motion variance, and skin tone variance. The health monitoring scenario investigates the feasibility of health monitoring at public spaces (i.e. airports, subways, malls). Five state of the art heart rate measuring methods were re-implemented and tested with the feasibility study database. The results were compared with ground truth to estimate the heart rate measurement error. The heart rate measurement error was analyzed using mean error, standard deviation; root means square error and Pearson correlation coefficient. The findings of this experiment inferred promising results for health monitoring of subjects standing at a distance of 500 cm.

Assertion-Based QA With Question-Aware Open Information Extraction

We present assertion based question answering (ABQA), an open domain question answering task that takes a question and a passage as inputs, and outputs a semi-structured assertion consisting of a subject, a predicate and a list of arguments. An assertion conveys more evidences than a short answer span in reading comprehension, and it is more concise than a tedious passage in passage-based QA. These advantages make ABQA more suitable for human-computer interaction scenarios such as voice-controlled speakers. Further progress towards improving ABQA requires richer supervised dataset and powerful models of text understanding. To remedy this, we introduce a new dataset called WebAssertions, which includes hand-annotated QA labels for 358,427 assertions in 55,960 web passages. To address ABQA, we develop both generative and extractive approaches. The backbone of our generative approach is sequence to sequence learning. In order to capture the structure of the output assertion, we introduce a hierarchical decoder that first generates the structure of the assertion and then generates the words of each field. The extractive approach is based on learning to rank. Features at different levels of granularity are designed to measure the semantic relevance between a question and an assertion. Experimental results show that our approaches have the ability to infer question-aware assertions from a passage. We further evaluate our approaches by incorporating the ABQA results as additional features in passage-based QA. Results on two datasets show that ABQA features significantly improve the accuracy on passage-based QA.