Human-computer Interaction Tasks Research Articles

Assessing mouse, trackball, touchscreen and leap motion in ship vibration conditions: A comparison of task performance, upper limb muscle activity and perceived fatigue and usability

A variety of interaction devices are increasingly used for human-computer interaction (HCI) tasks in ship vibration conditions, but have seldom been well assessed. This study aimed to examine task performance, upper limb muscle activity, and perceived fatigue and device usability for four typical interaction devices (i.e., mouse, trackball, touchscreen, and Leap Motion) under simulated ship vibration conditions. A two-factor within-subjects design was employed in this study, where participants performed basic HCI tasks with the four interaction devices under three conditions (i.e., static condition, and low and high vibration conditions). The results showed that vibration condition significantly reduced task performance, especially for Leap Motion. Differences in task performance, upper arm and shoulder muscle activities, perceived fatigue and device usability were found among interaction devices. Mouse and touchscreen achieved the best task performance, compared with trackball and Leap Motion, while both touchscreen and Leap Motion achieved larger upper limb muscle activities, compared with trackball and mouse. Relevance to industryThe findings provide important implications for the use and configuration of interaction devices, and for the development of prevention interventions for risks of musculoskeletal disorders in using interaction devices under ship vibration conditions.

Body-Area Capacitive or Electric Field Sensing for Human Activity Recognition and Human-Computer Interaction

Due to the fact that roughly sixty percent of the human body is essentially composed of water, the human body is inherently a conductive object, being able to, firstly, form an inherent electric field from the body to the surroundings and secondly, deform the distribution of an existing electric field near the body. Body-area capacitive sensing, also called body-area electric field sensing, is becoming a promising alternative for wearable devices to accomplish certain tasks in human activity recognition (HAR) and human-computer interaction (HCI). Over the last decade, researchers have explored plentiful novel sensing systems backed by the body-area electric field, like the ring-form smart devices for sign language recognition, the room-size capacitive grid for indoor positioning, etc. On the other hand, despite the pervasive exploration of the body-area electric field, a comprehensive survey does not exist for an enlightening guideline. Moreover, the various hardware implementations, applied algorithms, and targeted applications result in a challenging task to achieve a systematic overview of the subject. This paper aims to fill in the gap by comprehensively summarizing the existing works on body-area capacitive sensing so that researchers can have a better view of the current exploration status. To this end, we first sorted the explorations into three domains according to the involved body forms: body-part electric field, whole-body electric field, and body-to-body electric field, and enumerated the state-of-art works in the domains with a detailed survey of the backed sensing tricks and targeted applications. We then summarized the three types of sensing frontends in circuit design, which is the most critical part in body-area capacitive sensing, and analyzed the data processing pipeline categorized into three kinds of approaches. The outcome will benefit researchers for further body-area electric field explorations. Finally, we described the challenges and outlooks of body-area electric sensing, followed by a conclusion, aiming to encourage researchers to further investigations considering the pervasive and promising usage scenarios backed by body-area capacitive sensing.

UFace

Facial expression recognition (FER) is a crucial task for human-computer interaction and a multitude of multimedia applications that typically call for friendly, unobtrusive, ubiquitous, and even long-term monitoring. Achieving such a FER system meeting these multi-requirements faces critical challenges, mainly including the tiny irregular non-periodic deformation of emotion movements, high variability in facial positions and severe self-interference caused by users' own other behavior. In this work, we present UFace, a long-term, unobtrusive and reliable FER system for daily life using acoustic signals generated by a portable smartphone. We design an innovative network model with dual-stream input based on the attention mechanism, which can leverage distance-time profile features from various viewpoints to extract fine-grained emotion-related signal changes, thus enabling accurate identification of many kinds of expressions. Meanwhile, we propose effective mechanisms to deal with a series of interference issues during actual use. We implement UFace prototype with a daily-used smartphone and conduct extensive experiments in various real-world environments. The results demonstrate that UFace can successfully recognize 7 typical facial expressions with an average accuracy of 87.8% across 20 participants. Besides, the evaluation of different distances, angles, and interferences proves the great potential of the proposed system to be employed in practical scenarios.

Speech Emotion Recognition Using Convolution Neural Networks and Multi-Head Convolutional Transformer.

Speech emotion recognition (SER) is a challenging task in human-computer interaction (HCI) systems. One of the key challenges in speech emotion recognition is to extract the emotional features effectively from a speech utterance. Despite the promising results of recent studies, they generally do not leverage advanced fusion algorithms for the generation of effective representations of emotional features in speech utterances. To address this problem, we describe the fusion of spatial and temporal feature representations of speech emotion by parallelizing convolutional neural networks (CNNs) and a Transformer encoder for SER. We stack two parallel CNNs for spatial feature representation in parallel to a Transformer encoder for temporal feature representation, thereby simultaneously expanding the filter depth and reducing the feature map with an expressive hierarchical feature representation at a lower computational cost. We use the RAVDESS dataset to recognize eight different speech emotions. We augment and intensify the variations in the dataset to minimize model overfitting. Additive White Gaussian Noise (AWGN) is used to augment the RAVDESS dataset. With the spatial and sequential feature representations of CNNs and the Transformer, the SER model achieves 82.31% accuracy for eight emotions on a hold-out dataset. In addition, the SER system is evaluated with the IEMOCAP dataset and achieves 79.42% recognition accuracy for five emotions. Experimental results on the RAVDESS and IEMOCAP datasets show the success of the presented SER system and demonstrate an absolute performance improvement over the state-of-the-art (SOTA) models.

Intelligent Speech Emotion Classification Using Deep Learning Technique

Abstract: Speech emotion recognition is a very interesting but very challenging human-computer interaction task. In recent years, this topic has attracted a lot of attention. In the field of speech emotion recognition, many techniques have been used to extract emotion from signals, including many well-established speech analysis and classification techniques. In the traditional way of speech emotion recognition, the emotion recognition features are extracted from the speech signals, and then the features, which are collectively known as the selection module, are selected, and then the emotions are recognized, which is a very tedious and time-consuming process, so this paper provides an overview of the deep learning technique, which is based on a simple algorithm based on feature extraction and building a model that recognizes emotions

Few-shot learning for facial expression recognition: a comprehensive survey

Facial expression recognition (FER) is utilized in various fields that analyze facial expressions. FER is attracting increasing attention for its role in improving the convenience in human life. It is widely applied in human–computer interaction tasks. However, recently, FER tasks have encountered certain data and training issues. To address these issues in FER, few-shot learning (FSL) has been researched as a new approach. In this paper, we focus on analyzing FER techniques based on FSL and consider the computational complexity and processing time in these models. FSL has been researched as it can solve the problems of training with few datasets and generalizing in a wild-environmental condition. Based on our analysis, we describe certain existing challenges in the use of FSL in FER systems and suggest research directions to resolve these issues. FER using FSL can be time efficient and reduce the complexity in many other real-time processing tasks and is an important area for further research.

EM-Gaze: eye context correlation and metric learning for gaze estimation

In recent years, deep learning techniques have been used to estimate gaze—a significant task in computer vision and human-computer interaction. Previous studies have made significant achievements in predicting 2D or 3D gazes from monocular face images. This study presents a deep neural network for 2D gaze estimation on mobile devices. It achieves state-of-the-art 2D gaze point regression error, while significantly improving gaze classification error on quadrant divisions of the display. To this end, an efficient attention-based module that correlates and fuses the left and right eye contextual features is first proposed to improve gaze point regression performance. Subsequently, through a unified perspective for gaze estimation, metric learning for gaze classification on quadrant divisions is incorporated as additional supervision. Consequently, both gaze point regression and quadrant classification performances are improved. The experiments demonstrate that the proposed method outperforms existing gaze-estimation methods on the GazeCapture and MPIIFaceGaze datasets.

EEG-Based Multi-Class Emotion Recognition using Hybrid LSTM Approach

Emotion recognition is a crucial task in human-computer interaction, psychology, and neuroscience. Electroencephalogram (EEG)-based multi-class emotion recognition is a novel approach that aims to identify and classify human emotions by analysing EEG signals. Traditional methods of emotion recognition often face challenges in accurately identifying and classifying human emotions due to their complexity and subjectivity. EEG-based emotion recognition provides a direct and objective measure of three emotional states (positive, neutral, and negative), making it a promising tool for emotion recognition. The proposed hybrid LSTM approach combines the strengths of different traditional machine learning algorithms: Gaussian Naive Bayes (GNB), Support Vector Machine (SVM), Logistic Regression (LR), and Decision Tree (DT). The approach was tested on the EEG brainwave dataset, and LSTM achieved an accuracy of 95%, while the proposed hybrid LSTM-GNB, LSTM-SVM, LSTM-LR, and LSTM-DT models achieved 65%, 96%, 97%, and 96% accuracy, respectively. The contribution of this study is the development of a hybrid LSTM approach that combines the strengths of two different algorithms, resulting in higher accuracy for multi-class emotion recognition using EEG signals. The results demonstrate the potential of the hybrid LSTM approach for real-world applications such as emotion-based human-computer interaction and mental health diagnosis.

Self-supervised vision transformer-based few-shot learning for facial expression recognition

Facial expression recognition (FER) is embedded in many real-world human-computer interaction tasks, such as online learning, depression recognition and remote diagnosis. However, FER is often hindered by privacy concerns and low recognition accuracy due to inadequate data transfer restrictions on public clouds, insufficient quantities of effective labeled samples and class imbalance. To address the above challenges, we have developed an automatic privacy-preserving learning state recognition system for supervising the quality of online teaching with the cooperation of edge servers and cloud servers to reduce the risk of privacy exposure. In particular, we propose few-shot facial expression recognition with a self-supervised vision transformer (SSF-ViT) by integrating self-supervised learning (SSL) and few-shot learning (FSL) to train a deep learning model with fewer labeled samples. Specifically, a vision transformer (ViT) is jointly pretrained with four self-supervised pretext tasks, including image denoising and reconstruction, image rotation prediction, jigsaw puzzle and masked patch prediction, to obtain a pretrained ViT encoder. Then, the pretrained ViT encoder is used on a lab-controlled labeled FER dataset to extract the spatiotemporal features and implement the FER task to fine-tune the parameters. Finally, we construct prototypes to verify the few-shot classification method for specific expression recognition. Support and query sets are divided in the wild FER dataset, and few-shot classification episodes are constructed. The fine-tuned ViT encoder is used as the feature extractor to build the prototype for each support set category, and the expression classification results are obtained by computing the Euclidean distance between the query samples and the prototypes. The extensive experimental results show that SSF-ViT can achieve recognition accuracies of 74.95%, 66.04%, 63.69% and 90.98% on the FER2013, AffectNet, SFEW 2.0 and RAF-DB datasets, respectively. In addition, SSF-ViT can improve the recognition performance of specific expression categories on these datasets.

Intelligent hybrid hand gesture recognition system using deep recurrent neural network with chaos game optimisation

ABSTRACT Hand gesture recognition is considered an essential task in various human-computer interaction (HCI) applications. Therefore, developing a robust system for hand gesture recognition is a challenge. This work proposed a new hybrid approach named hybrid deep recurrent neural network (RNN) incorporated with a chaos game optimisation (CGO) algorithm for efficiently recognising hand gestures. The main objective of a hybrid recurrent neural network with chaos game optimisation (RNN-CGO) is to achieve the recognition of alphabet signs from 2D gesture images. The different stages included in this technique are pre-processing, feature extraction, feature selection, and classification. The proposed approach is executed using American Sign Language (ASL) dataset with the help of the PYTHON platform. The precision, accuracy, f1-score, and recall obtained for the proposed approach are 99.28%, 99.96%, 99.25%, and 99.28%, respectively. The inference time required for the proposed approach is 0.121 s. The proposed technique has less computational complexity. The results showed that the proposed approach is better than the existing hand gesture recognition model approaches.

A Task Modeling Method of Intelligent Human–Computer Interaction in Aircraft Cockpits Based on Information Load Flow

Defining how tasks should be performed is an essential prerequisite for designing the next-generation aircraft cockpits with multimodal intelligent human–computer interaction (HCI) modes. To this end, a task modeling method based on information load flow is proposed in this article, namely weighted social network analysis (WSNA). In the WSNA method, the node relation complexity in a weighted social network is quantified based on entropy, and composite matrices are established to describe the association frequency and strength between linked nodes. The performance evaluation indicators are proposed to measure the quality of weighted social networks on node centrality, node strength, network density, and network robustness, which contributes to identify the overload nodes and edges. The transformation rules from traditional modes to intelligent modes are also created as guidelines for application. A case study set in an HCI scenario during the approach phase is given for illustration. The information load and overall performance are compared in these two task modes. The results verify the feasibility of WSNA and thereby provide a solution to planning the HCI tasks in next-generation aircraft cockpits.

Research on the Applicability of Touchscreens in Manned/Unmanned Aerial Vehicle Cooperative Missions.

The suitability of touchscreens for human-computer interaction in manned/unmanned aerial vehicle cooperative missions remains uncertain, especially in situations that are time-sensitive with variations in difficulty levels. The purpose of this study is to determine the feasibility of touchscreen applications in manned/unmanned aerial vehicle cooperative missions and the magnitude of the effects of time pressure and task difficulty. In contrast to previous studies, a combination of performance and perceptual load measures was used to divide errors into disposition errors, undetected errors, and miscalculation errors to explore specific error mechanisms, set up typical manned/unmanned aerial vehicle cooperative human-computer interaction tasks, and set up antecedent features for potential factors. Thirty subjects participated in an experiment that required the use of touchscreens or keyboards to perform a human-computer interaction task in a simulated manned/unmanned aerial vehicle cooperative mission. Experiments were set at three task difficulties: low, medium, and high, and were matched to a set time pressure or no time pressure for two seconds for low difficulty, three seconds for medium difficulty, and four seconds for high difficulty. The results showed that the touchscreens improved the participants' response speed at a time pressure of 2 s or less compared with the use of a general input device; however, the task error rate also increased significantly. The higher the task difficulty was, the worse the performance was and the greater the perceived workload of the participants. The application of touchscreens in dynamic environments subjected the participants to greater physical demands. The performance of participants using a keyboard was no better than that when touchscreens were used during the experiment. Moreover, touchscreens did not significantly improve participant performance. The results support the possibility of using touchscreens in manned/unmanned aerial vehicle cooperative missions.

Research on Visual Awareness Interface Design Based on Human-Computer Interaction Task

By using the form and specific process of human-computer interaction to comprehensively understand and judge whether users can accurately and quickly recognize products, it has a positive effect on the current visual and perceptual interface design work. On the basis of understanding the achievements of current related scientific research projects, this paper analyzes how to construct and design a new visual and perceptual interface based on human-computer interaction tasks by studying the main factors that affect users’ cognition of visual information interface.

Effects of vibration and target size on the use of varied computer input devices in basic human‐computer interaction tasks

AbstractPointing and dragging are fundamental actions by input devices when interacting with computer graphical user interfaces (GUIs). Cockpits on modern vehicles have been increasingly equipped with GUIs, enabling pointing and dragging tasks to be frequently performed under vibration conditions. However, factors influencing these fundamental actions under vibration conditions have not been fully explored. This study aimed to explore the effects of vibration, input devices, and target size on the performance and perceived workload in basic human–computer interaction tasks. Twenty‐seven participants completed an experiment where they were required to conduct two pointing tasks and one dragging‐and‐dropping task using four input devices (mouse, touchscreen, trackball, and remote hand‐controller) under static and three vibration conditions (lateral, fore‐and‐aft, and omnidirectional vibration) with two target sizes (small and large). The results indicated that vibration caused longer task completion time, higher error rates, and more workload in completing pointing and dragging tasks. Both target size and input device affected task performance in vibration environments. Highest workload was perceived when using remote hand‐controller, followed by trackballs under all vibration conditions; there was no significant difference between mouse and touchscreen, except in terms of physical demand. The findings suggest that practitioners should fully consider the joint effects of input device, target size, and vibration to counteract adverse influence of vibration and optimize user interface designs of human‐computer systems in vibration environments.

Intermittent Control as a Model of Mouse Movements

We present Intermittent Control (IC) models as a candidate framework for modelling human input movements in Human–Computer Interaction (HCI). IC differs from continuous control in that users are not assumed to use feedback to adjust their movements continuously, but only when the difference between the observed pointer position and predicted pointer positions becomes large. We use a parameter optimisation approach to identify the parameters of an intermittent controller from experimental data, where users performed one-dimensional mouse movements in a reciprocal pointing task. Compared to previous published work with continuous control models, based on the Kullback–Leibler divergence from the experimental observations, IC is better able to generatively reproduce the distinctive dynamical features and variability of the pointing task across participants and over repeated tasks. IC is compatible with current physiological and psychological theory and provides insight into the source of variability in HCI tasks.

An FCNN-Based Super-Resolution Mmwave Radar Framework for Contactless Musical Instrument Interface

In this article, we propose a framework for contactless human-computer interaction (HCI) using novel tracking techniques based on deep learning-based super-resolution and tracking algorithms. Our system offers unprecedented high-resolution tracking of hand position and motion characteristics by leveraging spatial and temporal features embedded in the reflected radar waveform. Rather than classifying samples from a predefined set of hand gestures, as common in existing work on deep learning with mmWave radar, our proposed imager employs a regressive full convolutional neural network (FCNN) approach to improve localization accuracy by spatial super-resolution. While the proposed techniques are suitable for a host of tracking applications, this article focuses on their application as a musical interface to demonstrate the robustness of the gesture sensing pipeline and deep learning signal processing chain. The user can control the instrument by varying the position and velocity of their hand above the vertically-facing sensor. By employing a commercially available multiple-input-multiple-output (MIMO) radar rather than a traditional optical sensor, our framework demonstrates the efficacy of the mmWave sensing modality for fine motion tracking and offers an elegant solution to a host of HCI tasks. Additionally, we provide a freely available software package and user interface for controlling the device, streaming the data to MATLAB in real-time, and increasing accessibility to the signal processing and device interface functionality utilized in this article.

Development of adaptive human–computer interaction games to evaluate attention

AbstractIn this paper, several physical activity-based human–computer interaction (HCI) games which are developed and implemented for the improvement of attention, emotion, and sensory–motor coordination will be presented. The interface and the difficulty levels of these games are specially designed for the use of people with different age groups and disabilities. The games involve physical activities for the fulfillment of some basic HCI tasks which require hand and arm motion for control, such as fruit picking and air hockey, with adaptive difficulty levels based on varying parameters of the games and human performance. In the fruit picking game, several fruit images are moving from top to the bottom of the screen. Objective is to collect apples while avoiding the pears. The player’s hand will control the basket that collects the fruits. In the air hockey game, the player will try to score goals against computer-controlled opponent. The player’s hand will control the paddle to hit the puck to score or to defend his/her goal area. The player’s hand is recognized by Kinect RGB-D sensors in both games. Aim of the adaptive difficulty-based system is keeping the players engaged in the games. The games are tested with a group of deaf children (3.5–5 years) as a part of an ongoing project,1 to decrease the stress of the children and increase their positive emotions, attention, and sensory–motor coordination before the audiology tests. The game performances and the evaluation of the therapists show that the games have a positive impact on the children. The games are also tested with a group of adults as a control group, where a mobile EEG device is employed to detect the attention levels. For this purpose, the adults also attended a third game featuring a maze and controlled with Myo sensors.

Visual search in vibration environments: Effects of spatial ability, stimulus size and stimulus density

Visual search is one type of common encountered human-computer interaction tasks, but it has less been examined in vibration environments. The aim of this study was to examine the effects of one user characteristic (spatial ability) and two stimulus characteristics (i.e., stimulus size and stimulus density) on visual search performance during single- and double-target tasks in simulated vibration environments. Twenty-four participants attended an experiment where they were instructed to perform single- and double-target visual search tasks with varied levels of stimulus size and stimulus density under static, slight and moderate vibration environments, respectively. Results indicated that the two vibration conditions achieved comparable visual search performance, perceived visual fatigue, and perceived comfort with static condition across task types. Stimuli size and stimulus density yielded significant effects on visual search time in both single- and double-target tasks (p's < 0.001). Participants with high spatial ability were faster than those with low spatial ability in double-target tasks (p = 0.042). Users preferred interface design with medium-to-large stimulus size and low-to-medium stimulus density in visual search.

Parsing human image by fusing semantic and spatial features: A deep learning approach

How to parse the human image to obtain the text label corresponding to the human body is a critical task for human-computer interaction. Although previous methods have significantly improved the parsing performance, the problem of parsing confusion and tiny target missing remains unresolved, which leads to errors and incomplete inference accordingly. Targeting at these drawbacks, we fuse semantic and spatial features to mine the human body information based on the Dual Pyramid Unit convolutional neural network, named as DPUNet. DPUNet is composed of Context Pyramid Unit (CPU) and Spatial Pyramid Unit (SPU). Firstly, we design the CPU to aggregate the local to global semantic information, which exports the semantic feature for eliminating the semantic confusion. To capture the tiny targets for preventing the details from missing, the SPU is proposed to incorporate the multi-scale spatial information and output the spatial feature. Finally, the features of two complementary units are fused for accurate and complete human parsing results. Our approach achieves more excellent performance than the state-of-the-art methods on single human and multiple human parsing datasets. Meanwhile, the proposed framework is efficient with a fast speed of 41.2fps.

Facial Emotion Recognition Using Hybrid Features

Facial emotion recognition is a crucial task for human-computer interaction, autonomous vehicles, and a multitude of multimedia applications. In this paper, we propose a modular framework for human facial emotions’ recognition. The framework consists of two machine learning algorithms (for detection and classification) that could be trained offline for real-time applications. Initially, we detect faces in the images by exploring the AdaBoost cascade classifiers. We then extract neighborhood difference features (NDF), which represent the features of a face based on localized appearance information. The NDF models different patterns based on the relationships between neighboring regions themselves instead of considering only intensity information. The study is focused on the seven most important facial expressions that are extensively used in day-to-day life. However, due to the modular design of the framework, it can be extended to classify N number of facial expressions. For facial expression classification, we train a random forest classifier with a latent emotional state that takes care of the mis-/false detection. Additionally, the proposed method is independent of gender and facial skin color for emotion recognition. Moreover, due to the intrinsic design of NDF, the proposed method is illumination and orientation invariant. We evaluate our method on different benchmark datasets and compare it with five reference methods. In terms of accuracy, the proposed method gives 13% and 24% better results than the reference methods on the static facial expressions in the wild (SFEW) and real-world affective faces (RAF) datasets, respectively.