Computer Interaction Research Articles

Speech recognition and intelligent translation under multimodal human–computer interaction system

Abstract The traditional translation robot is limited to the translation of single-mode text images and text videos, which has the problem of low translation accuracy. Therefore, speech recognition and intelligent translation in multimodal human–computer interaction (HCI) system are proposed. First, the network structure of speech recognition model in multi-channel HCI system is established, and the multi-head self-attention mechanism is constructed. Then, the artificial intelligence voice wake-up function is designed, and a multimodal machine translation model is constructed. On this basis, selective attention is added to obtain visual recognition of perceived text, and the decoder is used for multimodal gating fusion to realize the output of encoder translation results. Experimental results show that this method has high BLUE value and high translation accuracy.

Recyclable and Stable Strain Sensors Based on Semi‐Wrapped Structure of Silver Nanowires in Polyvinyl Alcohol for Human Motion Monitoring

AbstractHighly sensitive strain sensors have been widely used in human motion monitoring, medical treatment, soft robots, and human–computer interaction, and the recycling of functional materials is in a huge demand for eco‐friendly and sustainable electronics. However, the manufacturing of recyclable strain sensors still remains challenging. Here, a semi‐wrapped structure based on silver nanowires and polyvinyl alcohol is proposed to realize a recyclable and stable strain sensor. It has shown excellent sensitivity, fast response, high stretchability and good environmental stability, and is successfully applied for human motion monitoring. In addition, a simple strategy is developed to effectively recycle silver nanowires in an eco‐friendly manner. The recyclable and stable strain sensor demonstrates potential applications in wearable and stretchable electronics, and the recycling strategy can be extended to other noble metal nanomaterials.

Coral-Inspired Capacitive Pressure Sensor with High Sensitivity and Wide Range for Human–Computer Interaction

Coral-Inspired Capacitive Pressure Sensor with High Sensitivity and Wide Range for Human–Computer Interaction

Cognitive evaluation based on regression and eye-tracking for layout on human–computer multi-interface

ABSTRACT The human–computer cooperation process guided by natural interaction, intelligent interaction, and human–computer integration is gradually becoming a new trend in human–computer interaction. Cooperative scenarios of human–computer interaction systems often contain multi-interface and multi-device results in edges often interrupt the cognitive ergonomics of interface layout. This research takes typical areas as an example to establish a stepwise regression model to predict reaction time at an arbitrary position on the left interface. It uses a foveal region to position the starting point of attention and a parafoveal region to calculate the radius of each objective area, and design 10 similar tasks to analyze eye-tracking indexes through physiological assessment. Unlike fixed thinking such as spatial proximity on multi-interfaces, this research summarises cognitive features of layout based on the positive and negative effects of edge impact through eye-tracking analysis. It analyzes cognition including input, process, and output in human–computer cooperation from human intelligence and artificial intelligence respectively, and visualises the mapping relationship between these indexes and specific stages of cognition. Besides, the quantitative evaluation of the regression equation and qualitative analysis of the eye-tracking indexes provide a reference for other interfaces around the front interface.

Speech emotion recognition for human–computer interaction

Speech emotion recognition for human–computer interaction

Text-guided image-to-sketch diffusion models

Recently, with the continuous advancement of deep learning techniques, research on sketch synthesis has been progressing. However, existing methods still face challenges in generating human-like freehand sketches from real-world natural images at both object and scene levels. To address this, we propose SketchDiffusion, a text-guided freehand sketch synthesis method based on conditional stable diffusion. In SketchDiffusion, we design a novel image enhancing module to efficiently extract high-quality image features. Moreover, we utilize additional guidance from global and local features extracted by a U-shaped diffusion guidance network to control the noise addition and denoising process of the diffusion model, thereby significantly improving controllability and performance in freehand sketch synthesis. Beyond the model architecture, we leverage the designed BLIP-based text generation method to create 70,280 text prompts for foreground, background, and panorama sketch synthesis in the extensive SketchyCOCO dataset, thereby improving the overall effectiveness of model training. Compared to the state-of-the-art methods, our proposed SketchDiffusion has shown an average improvement of over 16.4%, 16.75%, and 12.8% on three quantitative metrics (sketch recognition, sketch-based retrieval, and user perceptual study), respectively. Furthermore, our approach not only excels in synthesizing freehand sketches containing multiple abstract objects but also has multiple applications in supporting human–computer interaction.

Research on wheat broken rate and impurity rate detection method based on DeepLab-EDA model and system construction

The broken rate and impurity rate of wheat are important indicators for assessing the quality of combine harvester operations. In view of the overlapping, occlusion and dense adhesion between the scattered grains during the operation of the combine harvester, it is difficult to obtain the grain crushing characteristics and impurity mass, which leads to low detection accuracy. In this paper, a method for detecting wheat broken rate and impurity rate based on DeepLab-EDA semantic segmentation model was proposed, and a detection system was built. In the detection system, an image acquisition device was designed and developed based on the principle of electromagnetic vibration, and the deep learning model was deployed in the embedded processor. Through the human–computer interaction interface design, the online processing and analysis of wheat image data and the display of the detection results of broken rate and impurity rate were realized. Comparative experiments with traditional semantic segmentation models showed that the MIoU, MP and MR of the DeepLab-EDA model were 89.41 %, 95.97 % and 94.83 %, respectively, representing improvements of 9.94%, 7.41%, and 7.52% over the baseline model, and indicating a significant enhancement in the accurate identification and segmentation of broken grain and impurities. Based on this, indoor group matching experiments were conducted with three groups of broken rate and impurity rate levels set at 0.5%, 1.5%, and 2.5%, showing the average errors of 7.54% and 6.30% for broken rate and impurity rate detection systems, respectively. Furthermore, the detection device was installed under the grain outlet of the GM80 combine harvester for field experiments, which showed average errors of 13.32% and 9.77% for wheat broken rate and impurity rate, respectively. The effectiveness and accuracy of the wheat broken rate and impurity rate detection system meet the requirements, which can provide a data basis for intelligent control of combine harvester operation parameters by the operator.

Identification of Object Roughness Using a Flexible Capacitive 3D Force Transducer Featuring an Interlocking Microstructure

A flexible capacitive three-dimensional (3D) force transducer with four bottom electrodes is proposed. A dielectric layer with ionic effects and pyramid-like microstructures is designed to improve the sensing performance of this sensor. The arrayed sensor exhibits remarkable capabilities in capturing and decoupling 3D forces, showcasing a high sensitivity, a wide response range (0–30[Formula: see text]kPa), rapid response time (62[Formula: see text]ms), excellent repeatability under continuous pressure, as well as static and dynamic responses. By assembling a robotic arm to grasp various objects, the sensor demonstrates exceptional sensing capabilities for object roughness, thereby highlighting its potential application as a touch recognition system in human–computer interaction scenarios.

Human–Computer Interaction (HCI) Advances to Re-Contextualize Cultural Heritage toward Multiperspectivity, Inclusion, and Sensemaking

Human–Computer Interaction (HCI) Advances to Re-Contextualize Cultural Heritage toward Multiperspectivity, Inclusion, and Sensemaking

Conductive, biodegradable multi-array smart bandage for gesture recognition and motion detection

Triboelectric nanogenerator (TENGs), which are capable of capturing high-entropy energy dispersed in the environment and converting it into electricity, have received significant attention for their potential as power sources for sensing networks and wearable portable electronics. Conventional triboelectric materials have serious limitations, which can be overcome by using nanostructured materials. Conductivity and semiconductor nanomaterials have been extensively studied as active components for this type of application, but need to be coupled to an elastic matrix, in order to be functional in flexible technologies. In this study, we prepared MWCNTs/PEO (MCP) films with conductive and biodegradable properties. MCP can be used as conductive ink printed on flexible substrates to fabricate flexible circuits. MCP is utilized as a high-performance positively charged material to construct a TENG (KMCP-TENG) with a maximum power density of 593 mW/m2. Furthermore, MCP can be degraded and recycled, and the output performance of the KMCP-TENG prepared using recycled MCP reaches 90 % of the original TENG. Additionally, the multi-array smart bandage (MASB) is designed and fabricated based on KMCP-TENG, which exhibits high sensitivity and stability. This MASB offers broad applications in health monitoring, motion tracking and human–computer interaction.

Research on jujube storage and preservation system based on Internet of Things (IoT)

With the surge in Xinjiang jujube production, there has been a concurrent increase in storage losses, posing an ongoing challenge for farmers to preserve the freshness of the jujubes. To tackle this issue, a novel storage and freshness control system for jujubes, integrating Internet of Things technology, has been developed. The control system is centered around a programmable logic controller (PLC) and utilizes an HMI as the interface for human–computer interaction. It is structured with distinct layers, including equipment, transportation, and application layers. By integrating with the Internet of Things cloud platform, data transmission between the device layer and the cloud server is ensured through the use of the TCP/IP protocol over a 4G network. A mobile app enables remote control functionality for monitoring and managing the storage and preservation environment. Experimental results demonstrate the system's stable performance, enabling remote monitoring and convenient operation by management personnel during the jujube storage and preservation process. This contributes to enhancing automation levels in jujube storage, extending its storage cycle while maintaining quality standards, thus providing valuable insights into intelligent fruit and vegetable storage practices in Xinjiang.

Multihead-Res-SE Residual Network with Attention for Human Activity Recognition

Human activity recognition (HAR) typically uses wearable sensors to identify and analyze the time-series data they collect, enabling recognition of specific actions. As such, HAR is increasingly applied in human–computer interaction, healthcare, and other fields, making accurate and efficient recognition of various human activities. In recent years, deep learning methods have been extensively applied in sensor-based HAR, yielding remarkable results. However, complex HAR research, which involves specific human behaviors in varied contexts, still faces several challenges. To solve these problems, we propose a multi-head neural network based on the attention mechanism. This framework contains three convolutional heads, with each head designed using one-dimensional CNN to extract features from sensory data. The model uses a channel attention module (squeeze–excitation module) to enhance the representational capabilities of convolutional neural networks. We conducted experiments on two publicly available benchmark datasets, UCI-HAR and WISDM, to evaluate our model. The results were satisfactory, with overall recognition accuracies of 96.72% and 97.73% on their respective datasets. The experimental results demonstrate the effectiveness of the network structure for the HAR, which ensures a higher level of accuracy.

A dual-channel multimodal sentiment analysis framework based on three-way decision

In the exploration of human–computer interaction, accurately identifying and understanding the user’s sentimental state helps enhance the user experience. Multimodal sentiment analysis aims to identify sentiments through various signals. However, inconsistencies in sentiments between text and visual–audio signals pose a challenge, as models often struggle to capture these differences adequately. To alleviate this issue, we propose a dual-channel sentiment analysis framework based on three-way decision. This framework includes a multimodal model, a visual–audio model, and a three-way decision module. Specifically, the multimodal model and visual–audio model generate predictions, and the three-way decision module selects an appropriate prediction. We improve the binary cross-entropy loss function for the visual–audio model to integrate the three-way decision module better and improve sentiment prediction accuracy. Additionally, we introduce an adversarial training task to ensure differences in predictions between the two models, thereby enhancing the influence of the visual–audio model. Experimental results on four public datasets demonstrate that our framework significantly improves the accuracy of sentiment recognition and reduces the average decision time.

Knowledge Distillation for Enhanced Age and Gender Prediction Accuracy

In recent years, the ability to accurately predict age and gender from facial images has gained significant traction across various fields such as personalized marketing, human–computer interaction, and security surveillance. However, the high computational cost of the current models limits their practicality for real-time applications on resource-constrained devices. This study addressed this challenge by leveraging knowledge distillation to develop lightweight age and gender prediction models that maintain a high accuracy. We propose a knowledge distillation method using teacher bounds for the efficient learning of small models for age and gender. This method allows the student model to selectively receive the teacher model’s knowledge, preventing it from unconditionally learning from the teacher in challenging age/gender prediction tasks involving factors like illusions and makeup. Our experiments used MobileNetV3 and EfficientFormer as the student models and Vision Outlooker (VOLO)-D1 as the teacher model, resulting in substantial efficiency improvements. MobileNetV3-Small, one of the student models we experimented with, achieved a 94.27% reduction in parameters and a 99.17% reduction in Giga Floating Point Operations per Second (GFLOPs). Furthermore, the distilled MobileNetV3-Small model improved gender prediction accuracy from 88.11% to 90.78%. Our findings confirm that knowledge distillation can effectively enhance model performance across diverse demographic groups while ensuring efficiency for deployment on embedded devices. This research advances the development of practical, high-performance AI applications in resource-limited environments.

Evaluation of perceived urgency from single-trial EEG data elicited by upper-body vibration feedback using deep learning

Notification systems that convey urgency without adding cognitive burden are crucial in human–computer interaction. Haptic feedback systems, particularly those utilizing vibration feedback, have emerged as a compelling solution, capable of providing desirable levels of urgency depending on the application. High-risk applications require an evaluation of the urgency level elicited during critical notifications. Traditional evaluations of perceived urgency rely on subjective self-reporting and performance metrics, which, while useful, are not real-time and can be distracting from the task at hand. In contrast, EEG technology offers a direct, non-intrusive method of assessing the user’s cognitive state. Leveraging deep learning, this study introduces a novel approach to evaluate perceived urgency from single-trial EEG data, induced by vibration stimuli on the upper body, utilizing our newly collected urgency-via-vibration dataset. The proposed model combines a 2D convolutional neural network with a temporal convolutional network to capture spatial and temporal EEG features, outperforming several established EEG models. The proposed model achieves an average classification accuracy of 83% through leave-one-subject-out cross-validation across three urgency classes (not urgent, urgent, and very urgent) from a single trial of EEG data. Furthermore, explainability analysis showed that the prefrontal brain region, followed by the central brain region, are the most influential in predicting the urgency level. A follow-up neural statistical analysis revealed an increase in event-related synchronization (ERS) in the theta frequency band (4–7 Hz) with the increased level of urgency, which is associated with high arousal and attention in the neuroscience literature. A limitation of this study is that the proposed model’s performance was tested only the urgency-via-vibration dataset, which may affect the generalizability of the findings.

Structure-Activity Relationship of Truncated 4'-Selenonucleosides: A3 Adenosine Receptor Activity and Binding Selectivity.

This study explored the impact of structural modifications on truncated 4'-selenonucleosides as ligands for the A3 adenosine receptor (AR). We synthesized and evaluated a series of these compounds for their binding affinities, functional activities, and structural interactions by using computational modeling. The SAR study revealed that all compounds exhibited selective and notable hA3AR binding, among which 6l (K i = 5.2 nM) and 6m (K i = 5.7 nM) were found as the best binding compounds. The representative N 6-cyclopropyl compound 6m was found to be a partial agonist, contrasting with the antagonist profiles of truncated 4'-oxo and 4'-thionucleosides. Computational docking highlighted 6m's unique interaction with Thr94 at the A3AR binding site. This research not only advances our understanding of A3AR ligand interactions but also highlights the potential of truncated 4'-selenonucleosides as effective A3AR modulators.

Investigating the User Experience Design of Home Ventilators Through the Lens of Kansei Engineering

Abstract Obstructive sleep apnea hypopnea syndrome (OSAHS) is a disease characterized by respiratory disorders, and its prevalence rate is on the rise worldwide. Home ventilator can replace or enhance the respiratory function of human body and alleviate the symptoms of OSAHS, and has become an important treatment, rehabilitation, and nursing equipment for patients. However, the existing products lack humanistic care in emotional level, which leads to negative emotions in the process of human–computer interaction. This study seeks to enhance the user experience of home ventilators by applying the principles of Kansei Engineering (KE). Through a principal component analysis of nine pairs of Kansei image adjectives associated with home ventilators, we concluded that users prefer home ventilators that are “friendly,” “technological,” “pleasant ,” “orderly ,” and “easy to understand.” These findings were utilized in the Kansei design of home ventilators. The effectiveness of these conclusions in improving the user experience of ventilators was subsequently validated using the PrEmo method.

A Study of Model Iterations of Fitts’ Law and Its Application to Human–Computer Interactions

Fitts’ law, a predictive model for motor task completion time, is widely utilized in human–computer interaction (HCI) research. While its formulas in two dimensions have achieved consensus over the decades, research diverges on its application in three dimensions. This paper synthesizes practical applications across touchscreens, virtual reality (VR), pedals, handheld devices, etc., with a specific emphasis on enhancing interaction experiences for vulnerable populations. This review studies Fitts’ law’s applicability in diverse interaction scenarios, highlighting design considerations for touchscreens and handheld/foot-held devices. This article underscores the need for future research to explore three-dimensional applications and consider user age, with potential expansions into medical and sports domains. This systematic review aims to empower designers in crafting more ergonomic products and improving HCI experiences.

Skeuomorphic or flat? The effects of icon style on visual search and recognition performance

Although there have been many previous studies on icon visual search and recognition performance, only a few have considered the effects of both the internal and external characteristics of icons. In this behavioral study, we employed a visual search task and a semantic recognition task to explore the effects of icon style, semantic distance (SD), and task difficulty on users’ performance in perceiving and identifying icons. First, we created and filtered 64 new icons, which were divided into four different groups (flat design & close SD, flat design & far SD, skeuomorphic design & close SD, skeuomorphic design & far SD) through expert evaluation. A total of 40 participants (13 men and 27 women, ages ranging from 19 to 25 years, mean age = 21.9 years, SD=1.93) were asked to perform an icon visual search task and an icon recognition task after a round of learning. Participants’ accuracy and response time were measured as a function of the following independent variables: two icon styles (flat or skeuomorphic style), two levels of SD (close or far), and two levels of task difficulty (easy or difficult). The results showed that flat icons had better visual search performance than skeuomorphic icons; this beneficial effect increased as the task difficulty increased. However, in the icon recognition task, participants’ performance in recalling skeuomorphic icons was significantly better than that in recalling flat icons. Furthermore, a strong interaction effect between icon style and task difficulty was observed for response time. As the task difficulty decreased, the difference in recognition performance between these two different icon styles increased significantly. These findings provide valuable guidance for the design of icons in human–computer interaction interfaces.

Flexible sensor based on conformable, sticky, transparent elastomers for electronic skin

As 5G technology advances and enhances the interconnection of science and technology, the demand for advanced sensors has increased. Conventional rigid sensors are inadequate to meet these new requirements, thus sparking significant interest in flexible sensors. However, taking into account the biocompatibility, skin compliance, sensing performance, mechanical properties and optical properties of flexible sensors is still the direction of efforts. Therefore, in this study, the flexible sensor with a sandwich structure was prepared using AgNWs as a conductive filler and P32-PDMS elastomer as a flexible substrate. The PDMS elastomer is modified by blending PEG-PPG-PEG and PDMS, giving the P32-PDMS elastomer low Young’s modulus (32.35 kPa), high elongation at break (over 450 %) and good adhesion (18.94 N/m), and the P32-PDMS elastomer still retains a certain light transmittance (82.22 %) and biocompatibility (RGR=89.524). These properties simultaneously endow the flexible sensor with good sensitivity (GF=17.27), wide detection range (0–100 % strain), and excellent cycling stability (5000 cycles). When the flexible sensor is used as e-skin to monitor human body movements, stable signals are generated regardless of the low-strain movements of the human body (pulse and sound) or the high-strain movements of the human body (pressing and joint bending). Therefore, the prepared flexible sensor has great potential as e-skin in many fields such as personal health monitoring, human motion detection, human–computer interaction, flexible display, etc.