Interactive Computer System Research Articles

Research on click enhancement strategy of hand-eye dual-channel human-computer interaction system: Trade-off between sensing area and area cursor

This study aims to explore the application of click enhancement strategies in a “Sight Line Localization + Hand Triggered” eye-control human–computer interaction system, proposes a click enhancement strategy to solve two critical problems in eye-control human–computer interaction: Midas touch and low spatial accuracy. By conducting ergonomics experiments, we verify that the proposed click enhancement strategy can effectively improve the operational performance of hand-eye dual-channel HCI systems. The experimental results show that the accuracy of the operation can be significantly improved by using a cursor size equal to the diameter of the interaction control and a sensing area size of 1.8 times the diameter of the control. Based on the comprehensive consideration of operation efficiency and comfort, 0.75 times the control diameter of the cursor and 1.8 times the control diameter of the sensing area are the optimal parameter configurations. The results not only solve the problems of Midas touch and low spatial accuracy but also significantly reduce visual fatigue, thus improving the ease of use and robustness of the hand-eye dual-channel human–computer interaction system.

Artificial Intelligence, Virtual Computer Systems, and Interactive Systems: a Holistic Approach to Enhancing Creative Talent in Art Design

With the rapid development and wide application of artificial intelligence, virtual computer systems and interactive systems, they play an important role in improving innovative talents in art and design. The purpose of this paper is to explore the overall approach of using these technologies to improve innovation in the field of art and design as well as to cultivate innovative talents. Firstly, we introduce the current development of artificial intelligence in art and design. Secondly, we explore the application of AI, virtual computer systems and interactive systems in art and design innovation. Artificial intelligence technology can help designers discover new inspirations and ideas by analysing a large number of artworks and design cases; virtual computer systems can provide designers with a virtual creative environment so that they can experiment and create in an infinite space; interactive systems can provide learners with a personalised learning experience, and, through intelligent educational platforms and interactive tools, help them develop their creative thinking and problem-solving skills. By making full use of these technologies, we can cultivate more creative and innovative art and design talents, and promote the sustainable development of the art and design field.

Speech recognition and intelligent translation under multimodal human–computer interaction system

Abstract The traditional translation robot is limited to the translation of single-mode text images and text videos, which has the problem of low translation accuracy. Therefore, speech recognition and intelligent translation in multimodal human–computer interaction (HCI) system are proposed. First, the network structure of speech recognition model in multi-channel HCI system is established, and the multi-head self-attention mechanism is constructed. Then, the artificial intelligence voice wake-up function is designed, and a multimodal machine translation model is constructed. On this basis, selective attention is added to obtain visual recognition of perceived text, and the decoder is used for multimodal gating fusion to realize the output of encoder translation results. Experimental results show that this method has high BLUE value and high translation accuracy.

Cognitive evaluation based on regression and eye-tracking for layout on human–computer multi-interface

ABSTRACT The human–computer cooperation process guided by natural interaction, intelligent interaction, and human–computer integration is gradually becoming a new trend in human–computer interaction. Cooperative scenarios of human–computer interaction systems often contain multi-interface and multi-device results in edges often interrupt the cognitive ergonomics of interface layout. This research takes typical areas as an example to establish a stepwise regression model to predict reaction time at an arbitrary position on the left interface. It uses a foveal region to position the starting point of attention and a parafoveal region to calculate the radius of each objective area, and design 10 similar tasks to analyze eye-tracking indexes through physiological assessment. Unlike fixed thinking such as spatial proximity on multi-interfaces, this research summarises cognitive features of layout based on the positive and negative effects of edge impact through eye-tracking analysis. It analyzes cognition including input, process, and output in human–computer cooperation from human intelligence and artificial intelligence respectively, and visualises the mapping relationship between these indexes and specific stages of cognition. Besides, the quantitative evaluation of the regression equation and qualitative analysis of the eye-tracking indexes provide a reference for other interfaces around the front interface.

Emotion recognition using hierarchical spatial–temporal learning transformer from regional to global brain

Emotion recognition is an essential but challenging task in human–computer interaction systems due to the distinctive spatial structures and dynamic temporal dependencies associated with each emotion. However, current approaches fail to accurately capture the intricate effects of electroencephalogram (EEG) signals across different brain regions on emotion recognition. Therefore, this paper designs a transformer-based method, denoted by R2G-STLT, which relies on a spatial–temporal transformer encoder with regional to global hierarchical learning that learns the representative spatiotemporal features from the electrode level to the brain-region level. The regional spatial–temporal transformer (RST-Trans) encoder is designed to obtain spatial information and context dependence at the electrode level aiming to learn the regional spatiotemporal features. Then, the global spatial–temporal transformer (GST-Trans) encoder is utilized to extract reliable global spatiotemporal features, reflecting the impact of various brain regions on emotion recognition tasks. Moreover, the multi-head attention mechanism is placed into the GST-Trans encoder to empower it to capture the long-range spatial–temporal information among the brain regions. Finally, subject-independent experiments are conducted on each frequency band of the DEAP, SEED, and SEED-IV datasets to assess the performance of the proposed model. Results indicate that the R2G-STLT model surpasses several state-of-the-art approaches.

Design of Human–Computer Interaction Gesture Recognition System Based on a Flexible Biosensor

The continuous development of high-speed Internet technology has made the application of robots increasingly widespread. Current robots and human–computer interaction systems mostly use rigid materials, such as metals and semiconductors, which have limitations in terms of deformability and flexibility. In addition, the biocompatibility and user comfort of these materials are also an issue. Therefore, research into new flexible biosensors is essential to improve the flexibility, comfort, and interactivity of these systems. This research will select polymer hydrogel as the electrode material of the sensor and polydimethylsiloxane as the base material of the sensor to design a resistance flexible biosensor to solve the poor flexibility. The research will use a template-matching method to verify the feasibility of gesture recognition of the flexible sensor. The remote control system of the robot finger is designed by a proportional-integral differential controller tuned by aradial basis function neural network. The feasibility of the research system is verified by simulation and scene experiments. The flexible sensor studied and prepared had a sensitivity of 0.7269, a tensile limit of 300%, and a thickness of 0.16 mm, showing good sensitivity and stability. The recognition accuracy of the sensor designed in the study was 92.8%, which was 8.1% higher than that of the data glove. Compared with traditional proportional-integral derivative (PID) controllers, the improved controller system error was within 10 to 3 rad, which had better adaptability and stability. Key information includes the design method of the flexible biosensor, its high sensitivity and stability under multiple stretches, and the proposal and validation of a new RBFNN–PID control model. These results showed that using this new sensor and control model significantly improved the control accuracy of mechanical fingers and the effect of gesture recognition. These results have important implications for the development of more advanced human–computer interaction systems. They not only improve the performance and reliability of the system, but also improve the user's interactive experience. These technologies are particularly promising in the fields of prosthetics for disabled people, advanced game controllers, and remotely controlled robots operating in hazardous environments. The research results are expected to lead to the development of advanced prosthetics, augmented reality devices, advanced game controllers, and automated robots. The main contribution of the research is to design a resistive flexible biosensor, which improves the traditional sensor's poor flexibility and large size and improves the sensor's ability to sense small changes. Future research may focus on further improving the sensor's long-term stability and performance under a variety of environmental conditions. In addition, commercializing these technologies and making them universal is also an important direction for the future.

PACMHCI - Engineering Interactive Computing Systems, June 2024: Editorial Introduction

Welcome to this issue of the Proceedings of the ACM on Human-Computer Interaction, bringing together contributions from the community on Engineering Interactive Computing Systems (EICS). The EICS track of the PACM-HCI is the primary venue for research contributions at the intersection of Human-Computer Interaction (HCI) and Software Engineering.

An enhanced speech emotion recognition using vision transformer

In human–computer interaction systems, speech emotion recognition (SER) plays a crucial role because it enables computers to understand and react to users’ emotions. In the past, SER has significantly emphasised acoustic properties extracted from speech signals. The use of visual signals for enhancing SER performance, however, has been made possible by recent developments in deep learning and computer vision. This work utilizes a lightweight Vision Transformer (ViT) model to propose a novel method for improving speech emotion recognition. We leverage the ViT model’s capabilities to capture spatial dependencies and high-level features in images which are adequate indicators of emotional states from mel spectrogram input fed into the model. To determine the efficiency of our proposed approach, we conduct a comprehensive experiment on two benchmark speech emotion datasets, the Toronto English Speech Set (TESS) and the Berlin Emotional Database (EMODB). The results of our extensive experiment demonstrate a considerable improvement in speech emotion recognition accuracy attesting to its generalizability as it achieved 98%, 91%, and 93% (TESS-EMODB) accuracy respectively on the datasets. The outcomes of the comparative experiment show that the non-overlapping patch-based feature extraction method substantially improves the discipline of speech emotion recognition. Our research indicates the potential for integrating vision transformer models into SER systems, opening up fresh opportunities for real-world applications requiring accurate emotion recognition from speech compared with other state-of-the-art techniques.

Designable high-performance TPU foam strain sensors towards human-machine interfaces

Flexible, breathable, and stable sensors were important in developing human–computer interaction systems and novel wearable devices. However, major challenges were still faced in two aspects, 1) high-performance strain sensors with a large sensing range, and 2) an interaction mode that is both convenient and efficient. Herein, we developed a thermoplastic polyurethane large-skeleton foam (TPU-LS foam) through a steam-induced method. The strain sensor comprised three layers of breathable foam, with CNTs/rGO TPU-LS foam as the inner layer and pure foam as the protective layer. PDMS/TiO2 was sprayed to create a UV-resistant and self-repairing coating, achieving a 66% melanin inhibition rate and a 91% repair rate. The sensor presented a wide work range over 100%, a high gauge factor of 637.8, and long-term durability. We trained machine learning frameworks to recognize different gestures and designed two types of human–computer interaction applications, allowing users to operate apps with gestures.

Fuzzy Logic Method for Measuring Sustainable Decent Work Levels as a Corporate Social Responsibility Approach

The purpose of this study was to propose an interactive computer system that utilises the MATLAB Fuzzy Logic Designer to measure the level of implementation of SDG 8, which focuses on sustainable decent work (SDW) and economic growth. This study used policies and laws as parameters to determine the presence or absence of SDW. The fuzzy method was implemented in car windshield manufacturing in the auto parts industry as a case study to define and quantify work conditions and to determine the level of sustainable decent work (SDWL). The study described environmental conditions, such as noise, lighting, and heat stress; ergonomic factors, such as exposure time, the mass of the object manipulated, and lifting frequency; and organisation at work, such as workplace violence, salary, and workday, as linguistic variables. The level of the presence or absence of SDW was defined as their membership functions. The resulting vectors determined the absence of SDW with a score of 1.5 in two linguistic variables: environmental conditions and ergonomic factors. Some features of SDW in the linguistic variable organisation at work had an SDW score of 5. The SDWL vector determined a final score of 1.24, indicating the absence of decent work in production areas. This study found that the workers suffer a lack of long and healthy lives and a bad standard of living without economic growth due to work-related musculoskeletal disorders and work illnesses, increasing their out-of-pocket spending and catastrophic health expenses. As a CSR approach, assessing SDWLs helped managers improve policies and work conditions.

Identification Method of Unmanned Aerial Vehicle Graphical Control Strategy Based on Cloud Server

With the rapid development of unmanned aerial vehicle (UAV) technology, UAV has been widely used in agricultural plant protection, electric power inspection, security patrols, and other fields. However, the control system of the UAV is a complex human–computer interaction system, which requires higher requirements in practical applications. Due to differences in hardware design, software development, and other aspects among different manufacturers, these UAV control systems require high hardware requirements, resulting in a long development cycle. At the same time, in practical applications, due to various reasons, there are equipment failures that are difficult to detect and eliminate in a timely manner. This paper used the UAV graphical control strategy identification method based on cloud server technology, and used the support vector machine (SVM) algorithm to analyze its identification accuracy. The research results showed that when other conditions were the same, the number of researchers and experts who were satisfied with the drone trial effect of the cloud server was 42 and 10, respectively, accounting for 84% and 100%. It indicates that they believe that the cloud server can effectively improve the effectiveness of the drone graphical control strategy recognition method, indicating a positive relationship between the two.

Hardware and Software Design and Implementation of Surface-EMG-Based Gesture Recognition and Control System

The continuous advancement of electronic technology has led to the gradual integration of automated intelligent devices into various aspects of human life. Motion gesture-based human–computer interaction systems offer abundant information, user-friendly functionalities, and visual cues. Surface electromyography (sEMG) signals enable the decoding of muscle movements, facilitating the realization of corresponding control functions. Considering the inherent instability and minuscule nature of sEMG signals, this thesis proposes the integration of a dynamic time regularization algorithm to enhance gesture recognition detection accuracy and real-time system performance. The application of the dynamic time warping algorithm allows the fusion of three sEMG signals, enabling for the calculation of similarity between the sample and the model. This process facilitates gesture recognition and ensures effective communication between individuals and the 3D printed prosthesis. Utilizing this algorithm, the best feature model was generated by amalgamating six types of gesture classification model. A total of 600 training and evaluation experiments were performed, with each movement recognized 100 times. The experimental tests demonstrate that the accuracy of gesture recognition and prosthetic limb control using the temporal dynamic regularization algorithm achieves an impressive 93.75%, surpassing the performance of the traditional threshold control switch.

A wave-shaped electrode flexible sensor capable of sensitively responding to wrinkle excitation for a multifunctional human–computer interaction system

A wave-shaped electrode flexible sensor capable of sensitively responding to wrinkle excitation for a multifunctional human–computer interaction system

A Brain-Inspired Hierarchical Interactive In-Memory Computing System and Its Application in Video Sentiment Analysis

Video sentiment analysis can effectively establish the relationship between the emotion state and the multimodal information, while still suffer from intensive computation and low efficiency, due to the von Neumann computing architecture. Here, we present a brain-inspired hierarchical interactive in-memory computing (IMC) system, which can efficiently solve ‘von Neumann bottleneck’, enabling cross-modal interactions and semantic gap elimination. First, a 1T1M synapse array is fabricated using cost-effective, highly stable, flexible, and eco-friendly carbon materials, offering efficient analog multiply-accumulate operations. To illustrate the complexity of the proposed brain-inspired hierarchical interactive IMC system, three modules are proposed: 1) unimodal extraction module, 2) hierarchical interactive module, 3) output module. Furthermore, the proposed system is validated by applying it to video sentiment analysis. The experimental results demonstrate that the proposed system outperforms the existing state-of-the-art methods with high computational efficiency and good robustness. This work opens up a new way to achieve the deep integration of nanomaterials, deep learning, and modern electronics into IMC.

AIA-Net: Adaptive Interactive Attention Network for Text-Audio Emotion Recognition.

Emotion recognition based on text-audio modalities is the core technology for transforming a graphical user interface into a voice user interface, and it plays a vital role in natural human-computer interaction systems. Currently, mainstream multimodal learning research has designed various fusion strategies to learn intermodality interactions but hardly considers that not all modalities play equal roles in emotion recognition. Therefore, the main challenge in multimodal emotion recognition is how to implement effective fusion algorithms based on the auxiliary structure. To address this problem, this article proposes an adaptive interactive attention network (AIA-Net). In AIA-Net, text is treated as a primary modality, and audio is an auxiliary modality. AIA-Net adapts to textual and acoustic features with different dimensions and learns their dynamic interactive relations in a more flexible way. The interactive relations are encoded as interactive attention weights to focus on the acoustic features that are effective for textual emotional representations. AIA-Net performs well in adaptively assisting the textual emotional representation with the acoustic emotional information. Moreover, multiple collaborative learning (co-learning) layers of AIA-Net achieve multiple multimodal interactions and the deep bottom-up evolution of emotional representations. Experimental results on three benchmark datasets demonstrate the great effectiveness of the proposed method over the state-of-the-art methods.

STRATEGI PEMASARAN PRODUK PAKAIAN DENGAN SISTEM ONLINE DAN OFFLINE PADA TOKO KANAYA COLLECTION DI KOTA NEGERI LAMA LABUHAN BATU SUMATERA UTARA

Online and Offline Marketing Strategy is marketing carried out through an interactive online computer system, which connects consumers with sellers via the internet network, the internet is a term used to describe interacting with each other through social media accounts in the form of Facebook, making it possible to communicate. The research results show that: Online and Offline Marketing Strategy by collaborating and being able to increase sales at the Kanaya Collection Shop, Online Marketing Strategy by promoting products through social media accounts in the form of Facebook and can be maximized because it is able to bring in more consumers. and Offline Marketing Strategy, there is an increase in sales because consumers who come to the Kanaya Collection Shop can see directly the varied products and also affordable prices and strategic locations so that consumers are interested in shopping at the Kanaya Collection Shop.

EOG Based Eye Blink Detection using VB GUI for Eye Writing Applications

Usage of bio-signals from Electro-Oculo-Gram (EOG) in Human Computer Interaction (HCI) systems is one of the burgeoning study areas as of now. In order to provide support for physically challenged persons (or keypad Contactless HCI) in operating basic computer functions without making any physical touch with the computer keypad or mouse, we propose a framework utilization concept that changes movement of cursor on the computer screen using virtual keypad based on Visual Basic Graphical User Interface (VB GUI) with the eye movement detection through webcam of the system for different computer functions. In this work, VB GUI platform is used for a web browsing application and corresponding platform operation steps have been included. With the VB GUI, the basic cursor movements in the directions of left, right, up, and down to access the web links on the web page which is treated as eye writing in the concept of human computer interaction. Similarly, we can access different computer Apps such as MS-Doc, XL sheets, Audio and video files etc. Further, this paper focus is on prototype development of different home automation system applications.

Contactless human–computer interaction system based on three-dimensional holographic display and gesture recognition

Contactless human–computer interaction system based on three-dimensional holographic display and gesture recognition

SISTEM PENDUKUNG KEPUTUSAN UNTUK MENENTUKAN LOKASI PENDIRIAN HOTEL BARU BERBASIS ANALYTICAL HIERARCHY PROCESS (AHP) DI KABUPATEN KULON PROGO

Decision Support Systems or Decision Support Systems (DSS) are defined as interactive computer-based systems, which help decision makers to use data and models to solve unstructured problems. Decision Support Systems are interactive computer systems that can provide alternative solutions for decision makers, so decisions can be made more quickly and accurately.
 In an effort to speed up the decision-making process and avoid the subjectivity of the resulting decisions a decision support system is needed that aims to assist job management as decision makers in deciding the right location to establish new hotels through the selection process using the Analytical Hierarchy Process (AHP) model.
 This system is expected to assist management in comparing value criteria, applying pairwise comparison matrices to get the weight of each criterion both local and global criteria. The AHP method is chosen because this method can carry out multi-objective measurements and several criteria, according to the requirements in the decision for the establishment of new hotels that rely on criteria.

A Self‐Decoupled and High‐Resolution 2D Displacement Sensor Based on Triboelectric Nanogenerator for Manipulator Precision Positioning

Abstract2D displacement sensor (2DDS) is significant in precision manufacturing positioning, machine condition monitoring, robotic manipulation, and human–computer interaction system. However, the reported 2DDSs have severe limitations, such as low accuracy, poor decoupling, and lack of smart signal analysis. Herein, a self‐decoupled and high‐resolution 2DDS based on two freestanding triboelectric nanogenerators (F‐TENGs) is presented. This sensor achieves self‐decoupling by two orthogonal and independent 1D displacement sensors with low‐cost grating‐structured PCB. The test results of the sensor exhibit a high accuracy ≥99.06% (1D) and 98.94% (2D). Also, the experimental and simulation results demonstrate that this sensor has a high resolution of 200 µm and the function of initial position identification. A 2D displacement sensing system is developed that is capable of collecting F‐TENGs’ open‐circuit voltages in real‐time and performing 2D trajectory synthesis and display. This work expands the potential application of the TENG‐based 2DDS in robotic precision machining.