Intelligent Human-computer Interaction Research Articles

Bioinspired Three-Mode Photosensitive Synaptic LED for Optical Information Processing.

Inspired by human sense organs, AI is advancing toward multimodal perception, with display technology evolving into intelligent human-computer interaction tools. However, hardware networks with multimodal responses connected by different devices bring problems such as delayed information transfer and inefficiency. Thus, an innovative three-mode photosensitive synaptic LED (PSSL) is first proposed by adding a photosensitive layer indacenodithiophene-benzothiadiazole (IDTBT) to the quantum-dot light-emitting diode (QLED), switched by changing the bias voltage. The self-powered PSSL has a photoresponse range from 310 nm to 808 nm (ultraviolet-near-infrared, UV-NIR). The device exhibits a bipolar response under red and UV light at 1 V. When the voltage reaches the turn-on voltage, the PSSL device turns into a neuromorphic LED, exhibiting conductivity enhancement under red-light irradiation and suppression under UV-light irradiation. As a result, the PSSLs are expected to be applied in the field of optical encryption communication and in neuromorphic display.

Digital Human Intelligent Interaction System Based on Multimodal Pre-training Mode

ABSTRACT As the forefront of human–computer intelligent interaction, digital humans have increasingly diverse application scenarios, but also face many challenges. In order to enhance the intelligence level and interaction capability of digital human, the study first optimizes the traditional UniLM model, then introduces the mechanism of multi-head attention, and mitigates the exposure bias by using adversarial training and random replacement of decoder to design an improved UniLM model, and finally applies the improved model to the digital human management system. The results show that the precision rate of the improved UniLM model is improved by 7.68%, 6.4%, and 4.96%, the recall rate is improved by 11.94%, 9.69%, and 8.83%, and the F1-score is improved by 8.34%, 6.41%, and 7.68% compared with the other three models, which proves that it has a better precision rate, robustness, and generalization ability. The perplexity of the improved UniLM model is 135, 95, 76, 71, and 55 under five text lengths, which is significantly lower than the other models, proving that its text generation ability is better. The above results demonstrate the performance of the research-designed digital human system based on the Improved UniLM model, which provides a direction for the further development of digital human technology.

Interactive multi-agent convolutional broad learning system for EEG emotion recognition

Electroencephalogram (EEG) emotion recognition is gaining significance in intelligent human–computer interaction. Multi-agent learning can capture more complete and reliable features, compensating for the lack of a single agent’s knowledge. However, previous EEG emotion recognition only focused on a single agent. Therefore, to extract effective features from high-dimensional EEG data, a novel interactive multi-agent (IMA) learning framework is proposed, and introduced into convolutional broad learning system (CNNBLS), then the interactive multi-agent convolutional broad learning system (IMA-CNNBLS) is proposed for EEG emotion recognition. It can effectively model high-dimensional EEG data, automatically extract EEG features related to emotions through convolutional neural network (CNN), and then extend the above features to a vast space to quickly extract generalized features through broad learning system (BLS), consider a CNNBLS as an agent, and multi-agent interact in the IMA part. As the theoretical basis of IMA-CNNBLS, the consistency is mathematically proved through the stationary distribution theory of Markov processes. To demonstrate the superiority of our proposed model, sufficient experiments are conducted on the DEAP, DREAMER and SEED datasets. The experimental results show the model can effectively improve the EEG emotion recognition accuracy, the best recognition performance is achieved on all datasets. In addition, our proposed model also shows the multi-agent interaction significantly affects EEG emotion recognition.

A human activity recognition method based on Vision Transformer

Human activity recognition has a wide range of applications in various fields, such as video surveillance, virtual reality and human–computer intelligent interaction. It has emerged as a significant research area in computer vision. GCN (Graph Convolutional networks) have recently been widely used in these fields and have made great performance. However, there are still some challenges including over-smoothing problem caused by stack graph convolutions and deficient semantics correlation to capture the large movements between time sequences. Vision Transformer (ViT) is utilized in many 2D and 3D image fields and has surprised results. In our work, we propose a novel human activity recognition method based on ViT (HAR-ViT). We integrate enhanced AGCL (eAGCL) in 2s-AGCN to ViT to make it process spatio-temporal data (3D skeleton) and make full use of spatial features. The position encoder module orders the non-sequenced information while the transformer encoder efficiently compresses sequence data features to enhance calculation speed. Human activity recognition is accomplished through multi-layer perceptron (MLP) classifier. Experimental results demonstrate that the proposed method achieves SOTA performance on three extensively used datasets, NTU RGB+D 60, NTU RGB+D 120 and Kinetics-Skeleton 400.

An Embedded Electromyogram Signal Acquisition Device.

In this study, we design an embedded surface EMG acquisition device to conveniently collect human surface EMG signals, pursue more intelligent human-computer interactions in exoskeleton robots, and enable exoskeleton robots to synchronize with or even respond to user actions in advance. The device has the characteristics of low cost, miniaturization, and strong compatibility, and it can acquire eight-channel surface EMG signals in real time while retaining the possibility of expanding the channel. This paper introduces the design and function of the embedded EMG acquisition device in detail, which includes the use of wired transmission to adapt to complex electromagnetic environments, light signals to indicate signal strength, and an embedded processing chip to reduce signal noise and perform filtering. The test results show that the device can effectively collect the original EMG signal, which provides a scheme for improving the level of human-computer interactions and enhancing the robustness and intelligence of exoskeleton equipment. The development of this device provides a new possibility for the intellectualization of exoskeleton systems and reductions in their cost.

A light/thermal cascaded-driven equipment for machine recognition inspired by water lilies using as multifunctional soft actuator

Inspired by the natural stimulus–response of plants and animals, a range of soft equipment has been extensively developed and used in intelligent human–computer interaction, editable remote drives, and machine learning. Poly(N-isopropyl acrylamide) (PNIPAM) hydrogels have gained popularity thanks to their low phase transition temperature and excellent flexibility. However, the PNIPAM hydrogels suffer from poor mechanical properties, low electrical conductivity, and lack of self-responsiveness. Herein, a bilayer light/thermal cascaded-driven anisotropic poly acrylamide/poly(N-isopropylacrylamide-co-acrylamide)-polyvinyl alcohol-MXene (P/PP-M) actuator was prepared by a simple two-step polymerization method. The unique bionic light/thermal response mechanism enables rapid conversion of energy forms and the construction of remote editable actuators. Concretely, the mechanical properties of hydrogels (strain: 1014 %, stress: 94 KPa) have been greatly improved by copolymerizing MXene nanosheets and multiplex hydrogels through two-by-two interpenetrating molecular chains. MXene nanosheets formed dynamic networks conferring the hydrogel system with high conductivity and light/thermal conversion efficiency. The P/PP-M soft actuator features high sensitivity (Gauge factor = 3.62), fast response time (400 ms), and cycling durability (>500 cycles). Finally, the energy transfer mechanism from light to thermal energy to kinetic energy was corroborated by preparing a series of bionic equipment. Manual clips were prepared on this basis, which enables the transportation of objects and realizes the real-time reflection of the operating status of remotely driven soft manual clips through changes in electrical signals. Thus, the proposed efficient editable remote-driven machine recognition soft equipment provides new insights for developing intelligent flexible robots and wearable smart devices.

Intelligent Human-Computer Interaction: Combined Wrist and Forearm Myoelectric Signals for Handwriting Recognition.

Recent studies have highlighted the possibility of using surface electromyographic (EMG) signals to develop human-computer interfaces that are also able to recognize complex motor tasks involving the hand as the handwriting of digits. However, the automatic recognition of words from EMG information has not yet been studied. The aim of this study is to investigate the feasibility of using combined forearm and wrist EMG probes for solving the handwriting recognition problem of 30 words with consolidated machine-learning techniques and aggregating state-of-the-art features extracted in the time and frequency domains. Six healthy subjects, three females and three males aged between 25 and 40 years, were recruited for the study. Two tests in pattern recognition were conducted to assess the possibility of classifying fine hand movements through EMG signals. The first test was designed to assess the feasibility of using consolidated myoelectric control technology with shallow machine-learning methods in the field of handwriting detection. The second test was implemented to assess if specific feature extraction schemes can guarantee high performances with limited complexity of the processing pipeline. Among support vector machine, linear discriminant analysis, and K-nearest neighbours (KNN), the last one showed the best classification performances in the 30-word classification problem, with a mean accuracy of 95% and 85% when using all the features and a specific feature set known as TDAR, respectively. The obtained results confirmed the validity of using combined wrist and forearm EMG data for intelligent handwriting recognition through pattern recognition approaches in real scenarios.

MRSLN: A Multimodal Residual Speaker-LSTM Network to alleviate the over-smoothing issue for Emotion Recognition in Conversation

Multimodal Emotion Recognition in Conversation (ERC) plays a significant role in the field of human–computer intelligent interaction since it enables computers to perceive and infer the emotions expressed by the individuals, thereby intelligently responding to them. Most of current ERC methods pay more attention to modeling the complex interaction between different modalities. However, the features extracted by their unimodal networks are over-smoothed and may contain insufficient intra-speaker contextual information, which results in suboptimal results. In this paper, we focus on the unimodal learning and propose a simple late fusion framework named Multimodal Residual Speaker-LSTM Network (MRSLN), which uses speaker information to directly model inter-speaker and intra-speaker dependency, rather than fuse it into the learned features. MRSLN uses the speaker-LSTM consisting of the inter-speaker LSTM, intra-speaker LSTM, and the residual network between the input and output of the inter-speaker LSTM. Our proposed method can alleviate the issue of over-smoothing in deep Long Short Term Memory (LSTM) network and also incorporate additional intra-speaker contextual information. Extensive experiments conducted on IEMOCAP and MELD datasets demonstrate that MRSLN effectively captures inter-speaker and intra-speaker information and outperforms currently complex state-of-the-art (SOTA) models in efficiency and classification performance.

Research on Speech Communication Enhancement of English Web-based Learning Platform based on Human-computer Intelligent Interaction

This study presents a novel web-based learning platform that leverages human-computer intelligent interaction to enhance English communication skills. The platform integrates cutting-edge technologies to create an immersive learning experience, combining natural language processing, speech recognition, and interactive exercises. Learners engage in real-time conversations with virtual tutors, receive personalized feedback, and access a vast repository of educational resources. The platform not only facilitates language acquisition but also encourages self-paced learning, making it a valuable tool for both educators and students. By harnessing the power of artificial intelligence, this web-based platform represents a significant advancement in the realm of English language education. To overcome these issues this paper proposed SVM with an improved satin Bower bird optimization algorithm (SVM-ISBBO). SVM-ISBBO uses fog computing services that minimize the latency and speeds up the process, effectively handling huge wearable devices. In this proposed work SVM-ISBBO monitors the students communication, vocal parameters, blood pressure, etc, and these values are obtained from wearable sensor devices and their notifications are sent back to teachers. Teachers diagnosed the student information and sent back the alert notifications to the students for taking proper medications. All this information is stored in fog-based cloud storage in a secure manner. The accuracy rate of KNN got 78.56%, NB got 81.74%, SVM got 85.15% and the proposed work of SVM-ISBBO got 92.34%.

Current status and trends of technology, methods, and applications of Human–Computer Intelligent Interaction (HCII): A bibliometric research

Current status and trends of technology, methods, and applications of Human–Computer Intelligent Interaction (HCII): A bibliometric research

Convolutional Neural Networks‐Motivated High‐Performance Multi‐Functional Electronic Skin for Intelligent Human‐Computer Interaction

A multi-functional electronic skin (e-skin) with high sensitivity and wide response range is crucial for intelligent human-computer interaction (HCI) system. Here, a multi-functional hybrid e-skin (TPES) integrated by a piezo-capacitive pressure sensor (PCPS) and a triboelectric nanogenerator (TENG) is reported. An ion-gel electrospinning doped with MXene is combined with a micro-structured ion-gel to enrich the structural variations of the overall device, enabling TPES to achieve a high sensitivity of 616.42 kPa−1, a broad response range of 0.6 Pa–1 MPa, and a fast response time of 5.6 ms. The vast surface area of the mixed cellulose foam layer effectively enhances the electrical output of the TENG, making TPES’s application scenarios become more flexible. By further integrating the different functionalities of the TPES with one-dimensional convolutional neural networks and HCI module, a gesture recognition and material perception HCI system is realized. Additionally, benefiting from the excellent performance of the PCPS and TENG, a multi-information recognition HCI system is established, capable of simultaneously recognizing the pressure and surface material of the balls and enabling real-time interaction with a virtual character. In summary, the proposed work offers a novel avenue for practical intelligent HCI system in virtual reality field.

An Intelligent Human-Computer Interaction System Based on Wireless Self-Powered Sensor for Motion Monitoring

With the development of the Internet of Things and information technology, large numbers of self-powered flexible sensors are needed to monitor the state of human motions. Triboelectric nanogenerators (TENGs) have attracted attention in sports monitoring due to its low cost and materials versatility. An Eco-flex-PTFE TENG (PE-TENG) system has been designed to real-time monitor and analyze the position and force changes of soccer players when kicking the ball on the outside, inside, and front of the foot, and to evaluate the technical motions. The PE-TENG system includes a PE-TENG sensing unit, Bluetooth transmission end, and an upper computer digital signal receiving end. The friction layer of the PE-TENG sensing unit is composed of Eco-flex and PTFE. PE-TENG system has the advantages of small size, light weight, stable working performance, and does not require external power supply. On the basis of the PE-TENG system, designed a human-computer interaction system to achieve the combination of real and virtual kicking skills for analyzing soccer players’ kicking motions. The system provides a new direction for scientific monitoring of soccer training, and entertainment of soccer players, while also accelerating the broad application prospects of self-powered wearable sports sensors in the field of sports monitoring.

Advances and Challenges in Human Emotion Recognition Systems: A Comprehensive Review

It's really difficult to tell what a person is feeling simply by glancing at their face or their behaviour. A basic human quality, the ability to decipher nonverbal clues from body language and facial expressions is essential for social and everyday communication. People use voice, gestures, and emotions to communicate with one another. Thus, there is a high need in various industries for systems that can identify the same. In terms of artificial intelligence, if a computer can recognize and interpret human emotions, it will be much easier for it to engage with people. A number of methods have been proposed in the past for evaluating human emotion. The traditional techniques essentially use visual and auditory cues to simulate human emotional reactions, including speech, body language, and facial expressions. Characterizing emotional states by physiological reactions has garnered more attention in recent times. Rapid advances in technology should make it possible for complex and perceptive HCI (human-computer interaction) systems to consider emotional states of humans during interactions, promoting empathy between humans and machines. Intelligent Human-Computer Interaction (HCI) applications, including virtual reality, video games, and educational systems, need the ability to recognize emotions. In the medical domain, feelings that people identify with each other could be a sign of specific functional issues, such as severe depression. The primary goal of this review is to examine alternative methods for identifying emotions using five distinct approaches, rank and explain the best methods along with their benefits, and provide commentary. In an effort to improve human-computer interactions, this article intends to be a resource for all academics and students researching in the field of emotion detection.

Cartilage‐Inspired Multidirectional Strain Sensor with High Elasticity and Anisotropy Based on Segmented Embedded Strategy

AbstractFlexible, stretchable, and sensitive multidirectional sensing systems that can decouple different mechanical inputs and identify multidirectional signals are crucial for dynamic human signal perception and intelligent human–computer interaction. Most reported multidirectional sensors are suitable for discriminating in‐plane deformation directions, and the sensing materials are difficult to balance between stretchability and mechanical strength. Here, a segmented embedded structure strategy inspired by the interlaced structure of cartilage is proposed. This strategy combines soft and hard materials in a topological and zipper‐shear chain manner and balances the performance of reinforced composites with flexibility and high toughness. In the case of segmented embedded hydrogels (SEHs), a wearable multidirectional sensing system that can decouple and identify planar strain/pressure is constructed. The multidirectional sensing system exploits the inherent anisotropy and layered structure design of composites to decouple the sensing functions. Supported by machine learning algorithms, the high accuracy demonstration of the multidirectional sensors in typical multidirectional motion joint posture monitoring and recognition confirms their potential in practical applications such as personal health sensing and human–computer interaction.

Anisotropic Piezoresistive Sensors Made with Magnetically Induced Vertically Aligned Carbon Nanotubes/Polydimethylsiloxane.

A piezoresistive material consisting of internal vertically aligned carbon nanotubes acting in concert with an external microdome structure is prepared to obtain a flexible piezoresistive sensor with high anisotropy. Here, we first obtained flexible piezoresistive composites (VCP) with anisotropic properties by inducing the vertical alignment of multiwalled carbon nanotubes in the pressure direction under a weak magnetic field of 0.6 T. Then, the composite with a microdome structure on the surface (m-VCP) was fabricated by a mold with a microstructure to further increase the anisotropy of the composite. The m-VCP microstructure was docked with VCP and placed between two layers of copper foil. With the synergistic effect of vertically aligned carbon nanotubes and the microdome structure, the sensitivity of the flexible sensor in the pressure direction was dramatically increased. In the low-strain range (0-6%), the sensitivity of m-VCP (GF = 9.208) is improved by 49% compared to m-CP and by 86% compared to VCP. The sensor has high anisotropy in the piezoresistive direction and retains good fatigue resistance under fatigue testing for 2000 cycles. This means that the sensor can be used in emerging fields such as human health monitoring, wearable electronics, and intelligent human-computer interaction.

Special Issue on Deep Learning for Intelligent Human Computer Interaction

Special Issue on Deep Learning for Intelligent Human Computer Interaction

Super-stretching and high-performance ionic thermoelectric hydrogels based on carboxylated bacterial cellulose coordination for self-powered sensors

Self-powered sensors that do not require external power sources are crucial for next-generation wearable electronics. As environment-friendly ionic thermoelectric hydrogels can continuously convert the low-grade heat of human skin into electricity, they can be used in intelligent human-computer interaction applications. However, their low thermoelectric output power, cycling stability, and sensitivity limit their practical applications. This paper reports a 2,2,6,6-tetramethylpiperidine-1-oxyl radical (TEMPO)-oxidized carboxylated bacterial cellulose (TOBC) coordination double-network ionic thermoelectric hydrogel with lithium bis(trifluoromethane) sulfonimide (LiTFSI) as an ion provider for thermodiffusion, as LiTFSI exhibits excellent thermoelectric properties with a maximum power output of up to 538 nW at a temperature difference of 20 K. The interactions between the ions and the hydrogel matrix promote the selective transport of conducting ionic ions, producing a high Seebeck coefficient of 11.53 mV K−1. Hydrogen bonding within the polyacrylamide (PAAm) network and interactions within the borate ester bond within the TOBC confer excellent mechanical properties to the hydrogel such that the stress value at a tensile deformation of 3100 % is reaches 0.85 MPa. The combination of the high ionic thermovoltage and excellent mechanical properties ionic thermoelectric hydrogels provides an effective solution for the design and application of self-powered sensors based on hydrogels.

Human-computer interaction based on background knowledge and emotion certainty.

Aiming at the problems of lack of background knowledge and the inconsistent response of robots in the current human-computer interaction system, we proposed a human-computer interaction model based on a knowledge graph ripple network. The model simulated the natural human communication process to realize a more natural and intelligent human-computer interaction system. This study had three contributions: first, the affective friendliness of human-computer interaction was obtained by calculating the affective evaluation value and the emotional measurement of human-computer interaction. Then, the external knowledge graph was introduced as the background knowledge of the robot, and the conversation entity was embedded into the ripple network of the knowledge graph to obtain the potential entity content of interest of the participant. Finally, the robot replies based on emotional friendliness and content friendliness. The experimental results showed that, compared with the comparison models, the emotional friendliness and coherence of robots with background knowledge and emotional measurement effectively improve the response accuracy by 5.5% at least during human-computer interaction.

The Application of VR Technology in Environmental Design Based on Human-Computer Intelligent Interaction and Computing

Virtual reality technology makes environmental design more convenient, and the effect is more intuitive, so designers can create more diverse design works. The practical application of virtual reality technology can make the expression of environmental design extremely realistic, can make the public’s visual, auditory, and tactile feelings different to a certain extent, so that people can enjoy the immersive feeling brought by various virtual environments, and can also concentrate people’s attention to a certain extent, so that people’s impressions continue to deepen, and deepen the image of environmental art to a certain extent, so that it plays a role in beautifying the city. The continued maturity of virtual reality technology has made it widely used in many industries in society, as has environmental design expression. Virtual reality technology has a variety of characteristics, such as interactivity, conceptuality, and multidirectional perception. It breaks through the time and space limitations of traditional environmental design and can better display and highlight the design effect to customers.

GWrite: Enabling Through-the-Wall Gesture Writing Recognition Using WiFi

Recognizing in-air gestures can enable intelligent Human-Computer Interaction (HCI) applications and facilitate human lives. However, existing sensor/camera-based methods for gesture recognition are either non-ubiquitous, intrusive to privacy, or inconvenient to carry around. Contemporary device-free approaches require the person to be in the line of sight and proximity to the sensing device. This paper shows that WiFi signals can recognize hand-drawn in-air gestures even when the gesture location is non-line-of-sight/beyond walls to the WiFi transceivers. The proposed GWrite system utilizes the CSI time-series information from commercial WiFi chipsets. GWrite employs a unique approach for performing hand gestures, thus enabling the design of a hand movement model. Using the model and the time-reversal (TR) technique, this work derives a correspondence between the similarity of CSIs and the relative distance moved by the hand. This relation gave rise to unique features such as the number of segments, angle, and the intersection between segments that can classify a set of gesture shapes consisting of straight-line segments. GWrite achieved an accuracy of 92% on a group of 15 gestures. The proposed approach can be applied to a broader set of gestures, unlike the current systems that function over a limited gesture set.