Human-computer Interaction Systems Research Articles

Deep Learning Based Lipreading for Video Captioning

Visual speech recognition, often referred to as lipreading, has garnered significant attention in recent years due to its potential applications in various fields such as human-computer interaction, accessibility technology, and biometric security systems. This paper explores the challenges and advancements in the field of lipreading, which involves deciphering speech from visual cues, primarily movements of the lips, tongue, and teeth. Despite being an essential aspect of human communication, lipreading presents inherent difficulties, especially in noisy environments or when contextual information is limited. The McGurk effect, where conflicting audio and visual cues lead to perceptual illusions, highlights the complexity of lipreading. Human lipreading performance varies widely, with hearing-impaired individuals achieving relatively low accuracy rates. Automating lipreading using machine learning techniques has emerged as a promising solution, with potential applications ranging from silent dictation in public spaces to biometric authentication systems. Visual speech recognition methods can be broadly categorized into those that focus on mimicking words and those that model visemes, visually distinguishable phonemes. While word-based approaches are suitable for isolated word recognition, viseme-based techniques are better suited for continuous speech recognition tasks. This study proposes a novel deep learning architecture for lipreading, leveraging Conv3D layers for spatiotemporal feature extraction and bidirectional LSTM layers for sequence modelling. The proposed model demonstrates significant improvements in lipreading accuracy, outperforming traditional methods on benchmark datasets. The practical implications of automated lipreading extend beyond accessibility technology to include biometric identity verification, security surveillance, and enhanced communication aids for individuals with hearing impairments. This paper provides insights into the advancements, challenges, and future directions of visual speech recognition research, paving the way for innovative applications in diverse domains.

Hydrogel-based flexible degradable triboelectric nanogenerators for human activity recognition

The development of Internet of Things (IoT) and Artificial Intelligence (AI) technologies has led to an increased demand for wearable electronic devices that can be used in a variety of contexts. Hydrogel sensors have been regarded as an ideal material for the fabrication of multifunctional flexible wearable devices. However, it is difficult to create hydrogel sensors with ideal electrical and mechanical properties. In this study, a conductive hydrogel prepared by a simple method has been introduced. The hydrogel was immersed in a binary solvent of glycerol, water and sodium citrate to form a transparent, long-term stable and highly conductive gelatin-NaCl organohydrogel (GNOH). A stretchable and durable gelatin Ecoflex triboelectric nanogenerator (GE-TENG) was created by combining an organic hydrogel with Ecoflex elastomer. It has high electrical and mechanical performance, with an open-circuit voltage of 142 V, current of 3.8 μA, power output of 19.36 μW and response time of 85 ms. Moreover, the single-electrode mode of the GE-TENG provides high electrical output for powering portable electronic devices and can capture biomechanical energy in temperatures as low as −20 °C. As a result of these capabilities, GE-TENG can effectively identify human movements with precision and transmit signals wirelessly, thereby broadening its potential applications. The GE-TENG-based self-powered intelligent sensing system (GSIS) for human-computer interaction demonstrates a wearable intelligent system for controlling the movement of a tank model and a manipulator by integrating the signal acquisition, signal processing and signal output components. The system achieves 100% signal recognition accuracy when combined with covariance matrix feature analysis and Convolutional Neural Network classification algorithms. This research demonstrates the potential of wearable hydrogel-based electronic devices for monitoring human motion, harvesting biomechanical energy and human-computer interaction.

Ecofriendly and high-performance flexible pressure sensor derived from natural plant materials for intelligent audible and silent speech recognition

Flexible pressure sensors (FPSs) are widely used in human–computer interaction (HCI) and medical diagnoses, but their further development faces several challenges. Integrating artificial intelligence (AI) with FPSs is indispensable for advancing the cognitive capabilities of HCI systems, but the FPS must also be ecofriendly, cost-effective, and high-performance to meet the demands of environmental protection and sustainable development. Here, a biodegradable capacitive FPS is presented that is fabricated from natural plant materials (i.e., cornstarch and lignocellulose) while exhibiting remarkable attributes such as high sensitivity (2.427 kPa−1 at 0–2 kPa), rapid response/recovery time (20/24 ms), extremely low detection limit (∼10 Pa), outstanding stability (> 6000 fatigue tests), excellent biocompatibility (survival rate exceeding 99 % in cell viability assays), and prominent biodegradability (complete degradation within 24 h). Prepared sensors were integrated with optimized AI algorithms, embedded systems, and wireless transmission technologies to develop a dual-functional (i.e., audible and silent) speech recognition system that achieved recognition rates exceeding 95 %. The developed sensor is expected to play a crucial role in future HCI and medical assistance applications.

The Design of a Speech Recognition-Based Finger Training Rehabilitation Device: Mechanical and System Aspects

Aiming to address the extended diagnosis and recovery period along with low efficiency in traditional stroke patient rehabilitation, this study introduces a speech recognition-based finger rehabilitation training control system. This system enables patients to perform finger exercises while providing feedback on finger angle, speed, and position. Furthermore, it offers rehabilitation physicians valuable data for evaluation and reference in finger rehabilitation. The system is divided into hardware circuitry, a lower computer control system, and a voice recognition human-computer interaction system, all working in conjunction with the finger movement perception system. By applying the Hidden Markov Model (HMM) algorithm to the voice interaction system for pattern matching, the finger rehabilitation system undergoes simulation testing. The results demonstrate that the proposed rehabilitation training system based on speech recognition meets design requirements, ensuring safety, reliability, and substantial application potential in future finger rehabilitation training.

Study on Gesture Recognition Method with Two-Stream Residual Network Fusing sEMG Signals and Acceleration Signals.

Currently, surface EMG signals have a wide range of applications in human-computer interaction systems. However, selecting features for gesture recognition models based on traditional machine learning can be challenging and may not yield satisfactory results. Considering the strong nonlinear generalization ability of neural networks, this paper proposes a two-stream residual network model with an attention mechanism for gesture recognition. One branch processes surface EMG signals, while the other processes hand acceleration signals. Segmented networks are utilized to fully extract the physiological and kinematic features of the hand. To enhance the model's capacity to learn crucial information, we introduce an attention mechanism after global average pooling. This mechanism strengthens relevant features and weakens irrelevant ones. Finally, the deep features obtained from the two branches of learning are fused to further improve the accuracy of multi-gesture recognition. The experiments conducted on the NinaPro DB2 public dataset resulted in a recognition accuracy of 88.25% for 49 gestures. This demonstrates that our network model can effectively capture gesture features, enhancing accuracy and robustness across various gestures. This approach to multi-source information fusion is expected to provide more accurate and real-time commands for exoskeleton robots and myoelectric prosthetic control systems, thereby enhancing the user experience and the naturalness of robot operation.

A highly-sensitive wearable capacitance pressure sensor based on calcium copper titanate/polydimethysiloxane/graphene oxide and polydimethysiloxane/silver nanowires sanwich strustures combination for human body monitoring

In recent years, flexible pressure sensors, with their high sensitivity, low detection limit, wide working pressure range and fast response speed, have been playing an important role in the development of wearable artificial devices, human-computer interaction and healthcare systems, and have attracted wide attention. However, the pressure sensors face challenges related to the weak tightness of bipolar materials, low charge density, and limitation in dielectric layers. In this paper, we present a novel configuration of flexible capacitive pressure sensors, utilizing a polydimethysiloxane (PDMS) thin film covered by silver nanowires (PDMS/SN, PSN) as the electrode materials and a porous material composed of calcium copper titanate (@CCTO), PDMS and graphene oxide (GO)(@CCTO/PDMS/GO, @CPG) as the positive electrode material. The sacrificial material, sugar, is employed to create dense internal pores within the substrate material, @CPG, leading to the formation of a profoundly porous substrate. The integration of the flexible capacitive pressure sensor using @CPG/PSN results in enhanced output performance. The modified sensors achieves a sensitivity of 0.22 kPa−1, which is 7 times higher than that of a pure PDMS dielectric layer sensor. Moreover, the sensors has a response time of 87 ms, a low detection pressure of 5 Pa with an extensive detection range from 0 to 150 kPa, and are capable of enduring 7200 cycles without sustaining damage. To validate the practical performance of the @CPG/PSN sensor, various human body activities are detected, including swallowing, coughing, neck bending, wrist bending, elbow bending, finger flexing, and holding objects like cups using a digitally designed glove capable of accommodating different materials. The pressure sensor based on @CPG/PSN demonstrates significant promise in detecting both low and high pressures, thus expanding the potential applications of capacitive pressure sensors.

Product Design for the Elderly Based on Human-Computer Interaction in the Era of Big Data

Since 21st century, social aging trend has become more and more serious. It needs accurately identify the nursing needs of elderly disabled people, who are characterized by inconvenient movement and unclear speech. How to solve these problems has become a key point in the field of elderly care and medical care. In response to this problem, this research has designed a human-computer interactive gesture recognition system for elderly nursing beds in the context of big data. The recognition rate of the fusion feature + support vector machine (SVM) classifier adopted in this study is higher than 90% for each category of gesture. On the test set, this method has an average recognition rate of 96.35%, which is much higher than that of single feature + SVM classifier. While other methods’ recognition rate is lower than 90%. The recognition rate of tag c (nursing bed posture turning left) with obvious gesture feature information is as high as 99.28%, and that of tag h (nursing bed posture bedpan lowering) with weak gesture feature is 93.65%. The human-computer interaction system has well realized the recognition intention of user’s dynamic and static gestures, achieved the goal set by the research, and the interaction form is natural and reliable. In the later research, we can further realize a more comprehensive, accurate and natural human-computer interaction product design through the multi-channel joint decision-making scheme to meet the needs of the elderly.

Design and Optimization of Human-Computer Interaction System for Education Management Based on Artificial Intelligence

The continuous improvement and refinement of artificial intelligence (AI) technology has facilitated the broader application of human-computer interaction in the field of education management. The construction of an educational management human-computer interaction system based on AI technology can optimize and improve key parameters of educational management human-computer interaction scenarios, thereby creating a more comprehensive mobile learning (m-learning) application system. This paper is based on AI technology, analyzing gesture semantics and speech semantics, and combining fusion algorithms to construct an education management human interaction system. The performance changes of the system were compared with real experimental operations and the NOBOOK platform analysis. The results show that the education management human-computer interaction system constructed in this article can enhance the m-learning experience of participants. It ensures high recognition accuracy, leading to higher scores in all dimensions of indicator evaluation. Therefore, as one of the crucial forms of m-learning, the human-computer interaction system for education management based on AI can establish a foundation for the further enhancement and development of education management.

The Design of a Speech Recognition-Based Finger Training Rehabilitation Device: Mechanical and System Aspects

This research introduces a novel speech recognition-based control system for finger rehabilitation aimed at addressing the inefficiencies and prolonged recovery periods commonly associated with traditional stroke rehabilitation methods. The system enables stroke patients to engage in finger exercises while simultaneously providing real-time feedback on the finger's angle, speed, and position. This feedback is invaluable for rehabilitation physicians, offering critical data for assessing and refining rehabilitation strategies. The system architecture is composed of three main components: the hardware circuitry, a lower-level computer control system, and a voice recognition-based human-computer interaction system, all integrated with a finger movement perception system. Employing the Hidden Markov Model (HMM) for pattern recognition in the voice interaction component, the system has undergone simulation testing to verify its effectiveness. The findings confirm that the speech recognition-based rehabilitation training system meets all design expectations, providing a safe, reliable, and promising approach for enhancing finger rehabilitation practices.

Self-powered wearable human-computer interaction system based on kapok cellulose nanofibers

As wearable devices in the fields of medical health, human–computer interaction, and motion detection continue to diversify in their materials and forms, they show vast application potential. Cellulose materials are emerging as prominent materials for fabricating flexible wearable electronic devices because of their favorable skin compatibility, cost-effectiveness, and abundance. However, there is little research on the use of kapok cellulose in wearable electronics. This study addresses this knowledge gap by exploring the high-yield production and waste recycling potential of kapok cellulose. To overcome the limitations of conventional cellulose films, such as poor flexibility, high haze, and limited light transmittance, a kapok cellulose nanofiber film (KCNF) with a transparency of > 90 % and a tensile strain of > 20 % is prepared. The use of the KCNF as a friction electric material in the triboelectric nanogenerator (TENG) improves its electrical output performance by 228 % and permits its comfortable long-term attachment to the skin for self-powered tactile sensing applications, including game control, text input, and smart medical devices. This study demonstrates that the exploration of KCNF materials expands the options available for developing wearable electronic devices with numerous potential applications.

Optimization Analysis of AI Intelligent English Teaching Strategies Based on Computer Virtual Reality Technology

The Internet of Things and other technology breakthroughs have a big impact on how English is taught (IoTV). The investigation of the actual English teaching process and the identification of student features come first. In order to use the interpolation technique with the student image-based feature detection method, we investigate the IoTV reform. As a result, we find a clever algorithm for IoTV that recognises student features more effectively. Artificial intelligence (AI) is the design technique used to create the intelligence algorithm for pupil feature recognition. A comprehensive multifunctional human-computer interaction system utilizes a variety of input and output streams. In addition to the standard computer keyboard, cursor clicking, and screen touching, the most recent speech and facial recognition technology can be employed for data input. Students learned to orally interact with the robot and act as a guide to various destinations. The Multimodal Interaction System for English Education (MMIEE) is an investigation into the use of network and artificial intelligence (AI) in teaching. Recurrent neural network (Rein RNN)-based Reinforced learning is utilized for the perpectual evaluation model to categorize the questions posed by teachers in terms of their content and kind, and to conduct experimental investigation. The results show that the proposed Rein_RNN model achieves 98.6% of accuracy in 4.3sec of mean evaluation time.

Designable high-performance TPU foam strain sensors towards human-machine interfaces

Flexible, breathable, and stable sensors were important in developing human–computer interaction systems and novel wearable devices. However, major challenges were still faced in two aspects, 1) high-performance strain sensors with a large sensing range, and 2) an interaction mode that is both convenient and efficient. Herein, we developed a thermoplastic polyurethane large-skeleton foam (TPU-LS foam) through a steam-induced method. The strain sensor comprised three layers of breathable foam, with CNTs/rGO TPU-LS foam as the inner layer and pure foam as the protective layer. PDMS/TiO2 was sprayed to create a UV-resistant and self-repairing coating, achieving a 66% melanin inhibition rate and a 91% repair rate. The sensor presented a wide work range over 100%, a high gauge factor of 637.8, and long-term durability. We trained machine learning frameworks to recognize different gestures and designed two types of human–computer interaction applications, allowing users to operate apps with gestures.

Facial expression recognition with computer vision

Facial Expression Recognition (FER) is a specialized field within the domains of computer vision and pattern recognition, which is dedicated to the automated identification and examination of facial expressions. Facial expression recognition (FER) has attracted considerable scholarly interest in recent years owing to its diverse array of applications and its potential ramifications across multiple disciplines, such as psychology, human-computer interaction, marketing, and security systems. The objective of this study is to present a thorough examination of the scholarly progression of FER, elucidating the significant achievements, approaches, and obstacles encountered by researchers in this domain. The study presents a selection of databases that are appropriate for Facial Expression Recognition (FER) and conducts a comparative analysis of these databases. The primary methodologies are examined, and recommendations are provided for each stage. In conclusion, this research presents several suggestions for addressing both obstacles and potential in future research endeavors.

Identification Method of Unmanned Aerial Vehicle Graphical Control Strategy Based on Cloud Server

With the rapid development of unmanned aerial vehicle (UAV) technology, UAV has been widely used in agricultural plant protection, electric power inspection, security patrols, and other fields. However, the control system of the UAV is a complex human–computer interaction system, which requires higher requirements in practical applications. Due to differences in hardware design, software development, and other aspects among different manufacturers, these UAV control systems require high hardware requirements, resulting in a long development cycle. At the same time, in practical applications, due to various reasons, there are equipment failures that are difficult to detect and eliminate in a timely manner. This paper used the UAV graphical control strategy identification method based on cloud server technology, and used the support vector machine (SVM) algorithm to analyze its identification accuracy. The research results showed that when other conditions were the same, the number of researchers and experts who were satisfied with the drone trial effect of the cloud server was 42 and 10, respectively, accounting for 84% and 100%. It indicates that they believe that the cloud server can effectively improve the effectiveness of the drone graphical control strategy recognition method, indicating a positive relationship between the two.

Ululate: A Non-Intrusive, Wearable Tongue Gesture Detection System for Human-Computer Interaction

Ululate: A Non-Intrusive, Wearable Tongue Gesture Detection System for Human-Computer Interaction

3D Motion Gesture Control : Gesture Recognition and Adaptation for Human Computer Interaction

Advancements in human-computer interaction (HCI) have paved the way for more intuitive and immersive interfaces. The first part of the paper delves into the fundamental principles of 3D gesture recognition, including sensor technologies, machine learning algorithms, and computer vision techniques. It discusses the challenges associated with accurate recognition in various environmental conditions and the ways in which these challenges are being addressed by researchers. The second part focuses on the adaptation aspect of the technology. It highlights how 3D gesture recognition can be integrated into adaptive HCI systems, enabling personalized and context-aware interactions. These adaptations can range from adjusting the interface layout to suit the user's preferences to dynamically changing the system's behavior based on the user's gestures. Additionally, the paper discusses the potential applications of 3D gesture recognition in fields such as gaming, virtual reality, healthcare, and beyond. It emphasizes the need for continued research to improve accuracy, robustness, and user-friendliness, ultimately driving the widespread adoption of 3D gesture recognition in HCI.

Hardware and Software Design and Implementation of Surface-EMG-Based Gesture Recognition and Control System

The continuous advancement of electronic technology has led to the gradual integration of automated intelligent devices into various aspects of human life. Motion gesture-based human–computer interaction systems offer abundant information, user-friendly functionalities, and visual cues. Surface electromyography (sEMG) signals enable the decoding of muscle movements, facilitating the realization of corresponding control functions. Considering the inherent instability and minuscule nature of sEMG signals, this thesis proposes the integration of a dynamic time regularization algorithm to enhance gesture recognition detection accuracy and real-time system performance. The application of the dynamic time warping algorithm allows the fusion of three sEMG signals, enabling for the calculation of similarity between the sample and the model. This process facilitates gesture recognition and ensures effective communication between individuals and the 3D printed prosthesis. Utilizing this algorithm, the best feature model was generated by amalgamating six types of gesture classification model. A total of 600 training and evaluation experiments were performed, with each movement recognized 100 times. The experimental tests demonstrate that the accuracy of gesture recognition and prosthetic limb control using the temporal dynamic regularization algorithm achieves an impressive 93.75%, surpassing the performance of the traditional threshold control switch.

Convolutional Neural Networks‐Motivated High‐Performance Multi‐Functional Electronic Skin for Intelligent Human‐Computer Interaction

A multi-functional electronic skin (e-skin) with high sensitivity and wide response range is crucial for intelligent human-computer interaction (HCI) system. Here, a multi-functional hybrid e-skin (TPES) integrated by a piezo-capacitive pressure sensor (PCPS) and a triboelectric nanogenerator (TENG) is reported. An ion-gel electrospinning doped with MXene is combined with a micro-structured ion-gel to enrich the structural variations of the overall device, enabling TPES to achieve a high sensitivity of 616.42 kPa−1, a broad response range of 0.6 Pa–1 MPa, and a fast response time of 5.6 ms. The vast surface area of the mixed cellulose foam layer effectively enhances the electrical output of the TENG, making TPES’s application scenarios become more flexible. By further integrating the different functionalities of the TPES with one-dimensional convolutional neural networks and HCI module, a gesture recognition and material perception HCI system is realized. Additionally, benefiting from the excellent performance of the PCPS and TENG, a multi-information recognition HCI system is established, capable of simultaneously recognizing the pressure and surface material of the balls and enabling real-time interaction with a virtual character. In summary, the proposed work offers a novel avenue for practical intelligent HCI system in virtual reality field.

Research on the visualization of information of Chinese traditional music with human-computer interaction from the perspective of metaverse

<p>In the metaverse environment, establish an immersive human-computer interaction system for Chinese traditional music based on virtual reality technology. Design the system’s functionality according to the Y model, and construct a four-layered system architecture. Collect high-quality instructional audio and utilize polygon modeling technology to create contextualized scenes of Chinese traditional music, as well as high-fidelity models of characters and instruments. Implement motion capture through inertial sensor technology for performance action data mapping. Utilize a metaverse engine platform to realize interactive functions and conduct performance optimization. The system is capable of eliciting learners’ intrinsic experiences, enabling interactive self-directed learning and creative exploration of Chinese traditional music performance, demonstrating significant practical value.</p>

Humidity Adaptive Antifreeze Hydrogel Sensor for Intelligent Control and Human-Computer Interaction.

Conductive hydrogels have emerged as ideal candidate materials for strain sensors due to their signal transduction capability and tissue-like flexibility, resembling human tissues. However, due to the presence of water molecules, hydrogels can experience dehydration and low-temperature freezing, which greatly limits the application scope as sensors. In this study, an ionic co-hybrid hydrogel called PBLL is proposed, which utilizes the amphoteric ion betaine hydrochloride (BH) in conjunction with hydrated lithium chloride (LiCl) thereby achieving the function of humidity adaptive. PBLL hydrogel retains water at low humidity (<50%) and absorbs water from air at high humidity (>50%) over the 17 days of testing. Remarkably, the PBLL hydrogel also exhibits strong anti-freezing properties (-80°C), high conductivity (8.18 Sm-1 at room temperature, 1.9 Sm-1 at -80°C), high gauge factor (GF approaching 5.1). Additionally, PBLL hydrogels exhibit strong inhibitory effects against Escherichia coli (E. coli) and Staphylococcus aureus (S. aureus), as well as biocompatibility. By synergistically integrating PBLL hydrogel with wireless transmission and Internet of Things (IoT) technologies, this study has accomplished real-time human-computer interaction systems for sports training and rehabilitation evaluation. PBLL hydrogel exhibits significant potential in the fields of medical rehabilitation, artificial intelligence (AI), and the Internet of Things (IoT).