Human-computer Interaction Applications Research Articles

TAO

Translating fine-grained activity detection (e.g., phone ring, talking interspersed with silence and walking) into semantically meaningful and richer contextual information (e.g., on a phone call for 20 minutes while exercising) is essential towards enabling a range of healthcare and human-computer interaction applications. Prior work has proposed building ontologies or temporal analysis of activity patterns with limited success in capturing complex real-world context patterns. We present TAO, a hybrid system that leverages OWL-based ontologies and temporal clustering approaches to detect high-level contexts from human activities. TAO can characterize sequential activities that happen one after the other and activities that are interleaved or occur in parallel to detect a richer set of contexts more accurately than prior work. We evaluate TAO on real-world activity datasets (Casas and Extrasensory) and show that our system achieves, on average, 87% and 80% accuracy for context detection, respectively. We deploy and evaluate TAO in a real-world setting with eight participants using our system for three hours each, demonstrating TAO's ability to capture semantically meaningful contexts in the real world. Finally, to showcase the usefulness of contexts, we prototype wellness applications that assess productivity and stress and show that the wellness metrics calculated using contexts provided by TAO are much closer to the ground truth (on average within 1.1%), as compared to the baseline approach (on average within 30%).

Emotional Experience during Human-Computer Interaction: A Survey

As human-computer interaction (HCI) technology becomes more and more integrated into our daily life, increasing attention has been drawn towards the interaction experience in addition to HCI efficiency. In the present study, we conducted a survey to explore context-specific emotional experience in HCI. Four hundred participants were recruited to report the frequency of their emotional experiences on 44 fine-grained emotion items in six representative HCI scenarios. Compared with six matched human-human interaction (HHI) scenarios used as control, the HCI scenarios were in general more frequently associated with negative emotions, and less frequently associated with positive emotions, especially when computer served as a tool for communication with other people. Furthermore, the 44 emotional experience items in HCI were summarized as five factors, representing low-arousal focused, positively engaged, emotionally empathetic, high-arousal negative and frustratingly confused. Our study presents a comprehensive overview of context-specific emotional experience in human-computer interactions and provides a framework for emotion evaluation in HCI applications.

A BTO/PVDF/PDMS Piezoelectric Tangential and Normal Force Sensor Inspired by a Wind Chime.

There is a growing demand for flexible pressure sensors in environmental monitoring and human-robot interaction robotics. A flexible and susceptible sensor can discriminate multidirectional pressure, thus effectively detecting signals of small environmental changes and providing solutions for personalized medicine. This paper proposes a multidimensional force detection sensor inspired by a wind chime structure with a three-dimensional force structure to detect and analyze normal and shear forces in real time. The force-sensing structure of the sensor consists of an upper and lower membrane on a polydimethylsiloxane substrate and four surrounding cylinders. A piezoelectric hemisphere is made of BTO/PVDF/PDMS composite material. The sensor columns in the wind chime structure surround the piezoelectric layer in the middle. When pressure is applied externally, the sensor columns are connected to the piezoelectric layer with a light touch. The piezoelectric hemisphere generates a voltage signal. Due to the particular structure of the sensor, it can accurately capture multidimensional forces and identify the direction of the external force by analyzing the position of the sensor and the output voltage amplitude. The development of such sensors shows excellent potential for self-powered wearable sensors, human-computer interaction, electronic skin, and soft robotics applications.

Self-aware face emotion accelerated recognition algorithm: a novel neural network acceleration algorithm of emotion recognition for international students.

With an increasing number of human-computer interaction application scenarios, researchers are looking for computers to recognize human emotions more accurately and efficiently. Such applications are desperately needed at universities, where people want to understand the students' psychology in real time to avoid catastrophes. This research proposed a self-aware face emotion accelerated recognition algorithm (SFEARA) that improves the efficiency of convolutional neural networks (CNNs) in the recognition of facial emotions. SFEARA will recognize that critical and non-critical regions of input data perform high-precision computation and convolutive low-precision computation during the inference process, and finally combine the results, which can help us get the emotional recognition model for international students. Based on a comparison of experimental data, the SFEARA algorithm has 1.3× to 1.6× higher computational efficiency and 30% to 40% lower energy consumption than conventional CNNs in emotion recognition applications, is better suited to the real-time scenario with more background information.

A Deep Graph Network with Multiple Similarity for User Clustering in Human–Computer Interaction

User counterparts, such as user attributes in social networks or user interests, are the keys to more natural Human–Computer Interaction (HCI) . In addition, users’ attributes and social structures help us understand the complex interactions in HCI. Most previous studies have been based on supervised learning to improve the performance of HCI. However, in the real world, owing to signal malfunctions in user devices, large amounts of abnormal information, unlabeled data, and unsupervised approaches (e.g., the clustering method) based on mining user attributes are particularly crucial. This paper focuses on improving the clustering performance of users’ attributes in HCI and proposes a deep graph embedding network with feature and structure similarity (called DGENFS ) to cluster users’ attributes in HCI applications based on feature and structure similarity. The DGENFS model consists of a Feature Graph Autoencoder (FGA) module, a Structure Graph Attention Network (SGAT) module, and a Dual Self-supervision (DSS) module. First, we design an attributed graph clustering method to divide users into clusters by making full use of their attributes. To take full advantage of the information of human feature space, a k-neighbor graph is generated as a feature graph based on the similarity between human features. Then, the FGA and SGAT modules are utilized to extract the representations of human features and topological space, respectively. Next, an attention mechanism is further developed to learn the importance weights of different representations to effectively integrate human features and social structures. Finally, to learn cluster-friendly features, the DSS module unifies and integrates the features learned from the FGA and SGAT modules. DSS explores the high-confidence cluster assignment as a soft label to guide the optimization of the entire network. Extensive experiments are conducted on five real-world data sets on user attribute clustering. The experimental results demonstrate that the proposed DGENFS model achieves the most advanced performance compared with nine competitive baselines.

CLERA: A Unified Model for Joint Cognitive Load and Eye Region Analysis in the Wild

Non-intrusive, real-time analysis of the dynamics of the eye region allows us to monitor humans’ visual attention allocation and estimate their mental state during the performance of real-world tasks, which can potentially benefit a wide range of human-computer interaction (HCI) applications. While commercial eye-tracking devices have been frequently employed, the difficulty of customizing these devices places unnecessary constraints on the exploration of more efficient, end-to-end models of eye dynamics. In this work, we propose CLERA, a unified model for Cognitive Load and Eye Region Analysis, which achieves precise keypoint detection and spatiotemporal tracking in a joint-learning framework. Our method demonstrates significant efficiency and outperforms prior work on tasks including cognitive load estimation, eye landmark detection, and blink estimation. We also introduce a large-scale dataset of 30 k human faces with joint pupil, eye-openness, and landmark annotation, which aims at supporting future HCI research on human factors and eye-related analysis.

Flexible Wearable Strain Sensors Based on Laser-Induced Graphene for Monitoring Human Physiological Signals.

Flexible wearable strain sensors based on laser-induced graphene (LIG) have attracted significant interest due to their simple preparation process, three-dimensional porous structure, excellent electromechanical characteristics, and remarkable mechanical robustness. In this study, we demonstrated that LIG with various defects could be prepared on the surface of polyimide (PI) film, patterned in a single step by adjusting the scanning speed while maintaining a constant laser power of 12.4 W, and subjected to two repeated scans under ambient air conditions. The results indicated that LIG produced at a scanning speed of 70 mm/s exhibited an obvious stacked honeycomb micropore structure, and the flexible strain sensor fabricated with this material demonstrated stable resistance. The sensor exhibited high sensitivity within a low strain range of 0.4-8.0%, with the gauge factor (GF) reaching 107.8. The sensor demonstrated excellent stability and repeatable response at a strain of 2% after approximately 1000 repetitions. The flexible wearable LIG-based sensor with a serpentine bending structure could be used to detect various physiological signals, including pulse, finger bending, back of the hand relaxation and gripping, blinking eyes, smiling, drinking water, and speaking. The results of this study may serve as a reference for future applications in health monitoring, medical rehabilitation, and human-computer interactions.

Applications and challenges of large language models in critical care medicine

The rapid development of big data methods and technologies has provided more and more new ideas and methods for clinical diagnosis and treatment. The emergence of large language models (LLM) has made it possible for human-computer interactive dialogues and applications in complex medical scenarios. Critical care medicine is a process of continuous dynamic targeted treatment. The huge data generated in this process needs to be integrated and optimized through models for clinical application, interaction in teaching simulation, and assistance in scientific research. Using the LLM represented by generative pre-trained transformer ChatGPT can initially realize the application in the diagnosis of severe diseases, the prediction of death risk and the management of medical records. At the same time, the time and space limitations, illusions and ethical and moral issues of ChatGPT emerged as the times require. In the future, it is undeniable that it may play a huge role in the diagnosis and treatment of critical care medicine, but the current application should be combined with more clinical knowledge reserves of critical care medicine to carefully judge its conclusions.

Super-stretching and high-performance ionic thermoelectric hydrogels based on carboxylated bacterial cellulose coordination for self-powered sensors

Self-powered sensors that do not require external power sources are crucial for next-generation wearable electronics. As environment-friendly ionic thermoelectric hydrogels can continuously convert the low-grade heat of human skin into electricity, they can be used in intelligent human-computer interaction applications. However, their low thermoelectric output power, cycling stability, and sensitivity limit their practical applications. This paper reports a 2,2,6,6-tetramethylpiperidine-1-oxyl radical (TEMPO)-oxidized carboxylated bacterial cellulose (TOBC) coordination double-network ionic thermoelectric hydrogel with lithium bis(trifluoromethane) sulfonimide (LiTFSI) as an ion provider for thermodiffusion, as LiTFSI exhibits excellent thermoelectric properties with a maximum power output of up to 538 nW at a temperature difference of 20 K. The interactions between the ions and the hydrogel matrix promote the selective transport of conducting ionic ions, producing a high Seebeck coefficient of 11.53 mV K−1. Hydrogen bonding within the polyacrylamide (PAAm) network and interactions within the borate ester bond within the TOBC confer excellent mechanical properties to the hydrogel such that the stress value at a tensile deformation of 3100 % is reaches 0.85 MPa. The combination of the high ionic thermovoltage and excellent mechanical properties ionic thermoelectric hydrogels provides an effective solution for the design and application of self-powered sensors based on hydrogels.

Cross-user gesture recognition from sEMG signals using an optimal transport assisted student-teacher framework

The cross-user gesture recognition is a puzzle in the myoelectric control system, owing to great variability in muscle activities across different users. To address this problem, a novel optimal transport (OT) assisted student-teacher (ST) framework (termed OT-ST) was proposed in this paper to facilitate transfer across user domains in an unsupervised domain adaptation (UDA) manner. In this framework, the initial parameters of the ST models were trained with the labeled data from users in the source domain. In the model transfer stage for a new user in the target domain, the teacher model was utilized to generate pseudo labels for unlabeled testing samples, providing guidance to the adaptation of the student model. The OT algorithm was employed to optimize the pseudo labels generated from the teacher model, avoiding the model bias and further improving the effect of domain adaptation. The performance of the proposed OT-ST framework was evaluated via experiments of classifying seven hand gestures using high-density surface electromyogram (HD-sEMG) recordings from extensor digitorum muscles of eight intact-limbed subjects. The OT-ST framework yielded a high accuracy of 96.50 ± 2.88% for new users, and outperformed other common machine learning and UDA methods significantly (p < 0.01), demonstrating its effectiveness. The OT-ST framework does not require special repetitive training or any labeled data for calibration. In addition, it can incrementally learn from new testing samples and improve the recognition ability. This study provides a promising method for developing user-generic myoelectric pattern recognition, with wide applications in human-computer interaction, consumer electronics and prosthesis control.

Multiview Video-Based 3-D Hand Pose Estimation

Hand pose estimation (HPE) can be used for a variety of human-computer interaction applications such as gesture-based control for physical or virtual/augmented reality devices. Recent works have shown that videos or multi-view images carry rich information regarding the hand, allowing for the development of more robust HPE systems. In this paper, we present the Multi-View Video-Based 3D Hand (MuViHand) dataset, consisting of multi-view videos of the hand along with ground-truth 3D pose labels. Our dataset includes more than 402,000 synthetic hand images available in 4,560 videos. The videos have been simultaneously captured from six different angles with complex backgrounds and random levels of dynamic lighting. The data has been captured from 10 distinct animated subjects using 12 cameras in a semi-circle topology where six tracking cameras only focus on the hand and the other six fixed cameras capture the entire body. Next, we implement MuViHandNet, a neural pipeline consisting of image encoders for obtaining visual embeddings of the hand, recurrent learners to learn both temporal and angular sequential information, and graph networks with U-Net architectures to estimate the final 3D pose information. We perform extensive experiments and show the challenging nature of this new dataset as well as the effectiveness of our proposed method. Ablation studies show the added value of each component in MuViHandNet, as well as the benefit of having temporal and sequential information in the dataset. We make our dataset publicly available to contribute to the field at: <uri>https://github.com/LeylaKhaleghi/MuViHand</uri>.

Ecoflex Flexible Array of Triboelectric Nanogenerators for Gait Monitoring Alarm Warning Applications

The advent of self-powered arrays of tribological nanogenerators (TENGs) that harvest mechanical energy for data collection has ushered in a promising avenue for human motion monitoring. This emerging trend is poised to shape the future landscape of biomechanical study. However, when we try to monitor various regions of the foot across disparate environments simultaneously, it poses a number of problems, such as the lack of satisfactory waterproofing, suboptimal heat resistance, inaccurate monitoring capacity, and the inability to transmit data wirelessly. To overcome these issues, we have developed an array of sensors affixed to the insole’s surface to adeptly monitor movement gait patterns and alert users to falls using self-powered triboelectric nanogenerators (TENGs). Each sensor cell on this sensor works as an individual air gap TENG (FWF-TENG), namely flexible, waterproof, and fast response, composed of an Ecoflex single-electrode array. Each FWF-TENG boasts a fast response time of 28 ms, which is sufficient to quickly monitor pressure changes during various badminton activities. Importantly, these sensors can persistently generate electrical signals at 70%RH humidity. Data obtained from these sensors can be transmitted to an upper computer intelligent terminal wirelessly through multi-grouped FHW-ENG sensing terminals in real time to achieve human–computer interaction applications, including motion technical determinations, feedback, and fall alerts. As a result, the interconnected TENG arrays have broad potential applications, including gait rehabilitation monitoring, motion technique identification, and fall alarm applications.

EEG-based Emotion Recognition Using Sub-Band Time-Delay Correlations.

Emotion recognition is a challenging task with many potential applications in psychology, psychiatry, and human-computer interaction (HCI). The use of time-delay information in the controlled time-delay stability (cTDS) algorithm can help to capture the temporal dynamics of EEG signals, including sub-band information and bi-directional coupling that can aid in emotion recognition and identification of specific connectivity patterns between brain rhythms. Incorporating EEG frequency bands can be used to design better emotion recognition systems. This paper evaluates the cTDS algorithm for binary classification tasks of arousal and valence using EEG sub-band signals. This method achieved a high accuracy of 91.1% for arousal and 91.7% for valence based on one electrode recording site at Fp1. The cTDS algorithm is a promising approach to analyzing brain network interactions. It can be particularly applicable to arousal and valence classification tasks, especially within a complex, multimodal feature space associated with understanding psychiatric disorders and HCI applications.

One-piece knitted embedded hang-needle sensor for shoulder posture recognition

Wearable products have significant applications in medical assistance, posture correction, and humancomputer interaction. In this study, we present a knitting method for a one-piece knitted embedded hang-needle sensor. This method involves simultaneously knitting conductive yarn into the sensing zone while creating the monitoring garment in a seamless manner. By implementing a tight hanging needle structure in the sensing zone design and adjusting knitting parameters, the effect of pre-stretching on the sensor caused by the base garment is minimized. Furthermore, integrating the sensing area of the hang-needle structure with the monitoring garment eliminates the need for additional sensor integration steps and simplifies the preparation process, resulting in reduced time and material costs. Moreover, the monitoring garment offers a comfortable wearing experience without any foreign body sensation and accurately captures the initial shoulder posture of the wearer.

AutoML-Emo: Automatic Knowledge Selection Using Congruent Effect for Emotion Identification in Conversations

Emotion recognition in conversations (ERC) has wide applications in medical care, human-computer interaction, and other fields. Unlike the general task of emotion analysis, humans usually rely on context and commonsense knowledge to convey emotions in conversations. Only when the model can connect and fully utilize a large-scale commonsense knowledge base, it can better understand latent contents in conversations. Unfortunately, there is no available knowledge selection mechanism to address such knowledge needs and to make sure the system is not flooded with irrelevant commonsense knowledge. Therefore, we propose an AutoML strategy based on emotion congruent effect to select suitable knowledge and models, called AutoML-Emo. Global exploration and local exploitation-based selection mechanisms (G&LESM) are used for automatic knowledge selection. The transformer-based architecture search (TAS) is applied to model selection, the selected transformer-based model is employed to incorporate knowledge and capture context information in conversations. The experimental results show that AutoML-Emo can effectively enhance external knowledge in different sizes and domain datasets. Moreover, the selected transformer-based model derived from TAS is superior to the most advanced models.

Mussel-inspired cellulose nanofiber/poly(vinyl alcohol) hydrogels with robustness, self-adhesion and antimicrobial activity for strain sensors

Flexible strain sensors have attracted substantial attention given their application in human-computer interaction and personal health monitoring. Due to the inherent disadvantages of conventional hydrogels, the manufacture of hydrogel strain sensors with high tensile strength, excellent adhesion, self-healing and antimicrobial properties in vitro, and conductive stability is still a challenge. Herein, a conductive hydrogel consisting of polydopamine-coated cellulose nanofibers (CNF@PDA), carbon nanotubes (CNT), and polyvinyl alcohol (PVA) was developed. The CNTs in PVA/CNF@PDA/CNT hydrogels were uniformly dispersed in the presence of CNF@PDA by hydrogen bonding, resulting in a nearly threefold increase in conductivity (0.4 S/m) over hydrogels without PDA. The hydrogel exhibited satisfactory tensile properties (tensile stress up to 0.79 MPa), good fatigue resistance, self-recovery and excellent antimicrobial activity in vitro. It showed excellent adhesion, especially the adhesion strength of pigskin was increased to 27 kPa. In addition, the hydrogel was used as a strain sensor, exhibiting excellent strain sensitivity (strain coefficient = 2.29), fast response (150 ms), and great durability (over 1000 cycles). The fabricated strain sensors can detect both large and subtle human movements (e.g., wrist bending and vocalization) with stable and repeatable electrical signals, indicating potential applications in personal health monitoring.

Device-Free Tracking through Self-Attention Mechanism and Unscented Kalman Filter with Commodity Wi-Fi.

Recent advancements in target tracking using Wi-Fi signals and channel state information (CSI) have significantly improved the accuracy and efficiency of tracking mobile targets. However, there remains a gap in developing a comprehensive approach that combines CSI, an unscented Kalman filter (UKF), and a sole self-attention mechanism to accurately estimate the position, velocity, and acceleration of targets in real-time. Furthermore, optimizing the computational efficiency of such approaches is necessary for their applicability in resource-constrained environments. To bridge this gap, this research study proposes a novel approach that addresses these challenges. The approach leverages CSI data collected from commodity Wi-Fi devices and incorporates a combination of the UKF and a sole self-attention mechanism. By fusing these elements, the proposed model provides instantaneous and precise estimates of the target's position while considering factors such as acceleration and network information. The effectiveness of the proposed approach is demonstrated through extensive experiments conducted in a controlled test bed environment. The results exhibit a remarkable tracking accuracy level of 97%, affirming the model's ability to successfully track mobile targets. The achieved accuracy showcases the potential of the proposed approach for applications in human-computer interactions, surveillance, and security.

Counterfactual Scenario-relevant Knowledge-enriched Multi-modal Emotion Reasoning

Multi-modal video emotion reasoning (MERV) has recently attracted increasing attention due to its potential application in human-computer interaction. This task needs to not only recognize utterance-level emotions for conspicuous speakers, but also perceive the emotions of non-speakers in videos. Existing methods focus on modeling multi-modal multi-level contexts to capture emotion-relevant clues from the complex scenarios in videos. However, the context information is far from enough to infer the emotion labels of non-speakers due to the large gap between the scenario situation and emotions labels. Inspired by the observation that humans can find solutions to complex problems with the leverage of experience and knowledge, we propose SK-MER , a Scenario-relevant Knowledge-enhanced Multi-modal Emotion Reasoning framework for MERV task, which can leverage external knowledge to enhance the video scenario understanding and emotion reasoning. Specifically, we use scenario concepts extracted from videos to build knowledge subgraphs from external knowledge bases. The knowledge subgraphs are then utilized to obtain scenario-relevant knowledge representations through dynamic knowledge graph attention. Next, we incorporate the knowledge representations into context modeling to enhance emotion reasoning with external scenario-relevant knowledge. In addition, we propose a counterfactual knowledge representation learning approach to obtain more effective scenario-relevant knowledge representations. Extensive experimental results on MEmoR dataset show that the proposed SK-MER framework achieves new state-of-the-art results.

The use of human-computer interaction in libraries: a systematic literature review

Purpose This study aims to analyze the status and application/use of human–computer interaction (HCI) in libraries by conducting a systematic literature review (SLR). Design/methodology/approach A Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) approach was used to search Scopus, Web of Science and Google Scholar databases. The search criteria included research studies published in English language between 2010 and 2021, which were 4,167 citations. Out of 4,167 citations, a total of 50 studies were selected for the final analysis. Findings The results showed a positive attitude of librarians toward HCI applications in libraries worldwide. The results depict that one-third (30%) of the studies were conducted in the USA, followed by four (8%) studies in China. Out of 50 studies, a portion of 15 (30%) studies were based on digital libraries, followed by seven (14%) studies on academic libraries and five (10%) studies on libraries and their websites. HCI was used for searching and retrieving information, users’ interaction, authentication, online help/support, feedback, library web access, web OPAC, virtual access to resources, indigenous repository and virtual services. The most productive year was 2015, and journal of The Electronic Library had more articles on HCI than other journals. Practical implications The findings of this study could assist policymakers and library authorities in reconciling the HCI application in libraries for providing effective and efficient access and services to end-users. Originality/value This study is unique as no comprehensive study has been conducted on the use of HCI in librarianship using the SLR method.

Self-Adhesive, Anti-Freezing MXene-Based Hydrogel Strain Sensor for Motion Monitoring and Handwriting Recognition with Deep Learning.

Flexible strain sensors based on self-adhesive, high-tensile, super-sensitive conductive hydrogels have promising application in human-computer interaction and motion monitoring. Traditional strain sensors have difficulty in balancing mechanical strength, detection function, and sensitivity, which brings challenges to their practical applications. In this work, the double network hydrogel composed of polyacrylamide (PAM) and sodium alginate (SA) was prepared, and MXene and sucrose were used as conductive materials and network reinforcing materials, respectively. Sucrose can effectively enhance the mechanical performance of the hydrogels and improve the ability to withstand harsh conditions. The hydrogel strain sensor has excellent tensile properties (strain >2500%), high sensitivity with a gauge factor of 3.76 at 1400% strain, reliable repeatability, self-adhesion, and anti-freezing ability. Highly sensitive hydrogels can be assembled into motion detection sensors that can distinguish between various strong or subtle movements of the human body, such as joint flexion and throat vibration. In addition, the sensor can be applied in handwriting recognition of English letters by using the fully convolutional network (FCN) algorithm and achieved the high accuracy of 98.1% for handwriting recognition. The as-prepared hydrogel strain sensor has broad prospect in motion detection and human-machine interaction, which provides great potential application of flexible wearable devices.