Intelligent Human-computer Interaction Research Articles

Enterprise Strategic Management From the Perspective of Business Ecosystem Construction Based on Multimodal Emotion Recognition.

Emotion recognition (ER) is an important part of building an intelligent human-computer interaction system and plays an important role in human-computer interaction. Often, people express their feelings through a variety of symbols, such as words and facial expressions. A business ecosystem is an economic community based on interacting organizations and individuals. Over time, they develop their capabilities and roles together and tend to develop themselves in the direction of one or more central enterprises. This paper aims to study a multimodal ER method based on attention mechanism. It analyzes the current emotional state of consumers and the development direction of enterprises through multi-modal ER of human emotions and analysis of market trends, so as to provide the most appropriate response or plan. This paper firstly describes the related methods of multimodal ER and deep learning in detail, and briefly outlines the meaning of enterprise strategy in the business ecosystem. Then, two datasets, CMU-MOSI and CMU-MOSEI, are selected to design the scheme for multimodal ER based on self-attention mechanism. Through the comparative analysis of the accuracy of single-modal and multi-modal ER, the self-attention mechanism is applied in the experiment. The experimental results show that the average recognition accuracy of happy under multimodal ER reaches 91.5%.

ATDA: Attentional temporal dynamic activation for speech emotion recognition

Speech emotion recognition (SER) plays a vital role in intelligent human–computer interaction (HCI). The Convolutional Neural Network (CNN) is widely used in SER, effectively capturing the static local features but ignoring the temporal dynamic features necessary for SER. To solve this problem, we place an Attentional Temporal Dynamic Activation (ATDA) module into the CNN-based model to empower it to learn the static and dynamic features simultaneously. In particular, the ATDA module comprises a Temporal Dynamic Activation (TDA) block followed by a Multi-view and Multi-granularity Attention (MMA) block. The TDA block calculates the temporal difference at the feature level to activate the dynamic information and generate the fundamental dynamic feature. The MMA block further detects and amplifies the emotion-related dynamic features based on multiple attention views and granularities. These two blocks within the ATDA module cooperate to activate and extract the dynamic emotional features. Meanwhile, the static features are obtained by a convolutional layer, which are then combined with the dynamic features to generate the final emotional representations. Finally, experiments on the IEMOCAP, MSP-IMPROV, and MELD datasets reveal that the proposed ATDA-CNN model achieves competitive results and enhances SER accuracy by learning meaningful emotional representations.

Medical chatbot using AI and NLP

The use of chatbots has grown rapidly across industries, including marketing, assistive systems, education, healthcare, cultural heritage, and entertainment. This paper discusses the incentives for using chatbots and explains how useful chatbots are in various contexts. As intelligent software and hardware, also known as intelligent agents, are developed and analyzed, Artificial Intelligence (AI) is becoming more and more integrated into daily lives. From manual labor to complex procedures, intelligent agents are capable of performing a wide range of tasks. One of the simplest and most common forms of intelligent human-computer interaction is the chatbot, which is a classic example of an artificial intelligence Human-Computer Interaction (HCI) system. A chatbot is described as "a computer program designed to simulate interaction with human users, particularly over the Internet." In addition to chatbots, it also called smart bots, interactive agents, digital assistants, and intelligent conversational objects. In the midst of the COVID-19 pandemic, going to the doctor is no longer an indulgence. A chatbot is a Natural Language Processing (NLP) based chatbot to help with basic medical questions. Only the best knowledge of a chatbot can be used to answer medical questions.

DLD: An Optimized Chinese Speech Recognition Model Based on Deep Learning

Speech recognition technology has played an indispensable role in realizing human‐computer intelligent interaction. However, most of the current Chinese speech recognition systems are provided online or offline models with low accuracy and poor performance. To improve the performance of offline Chinese speech recognition, we propose a hybrid acoustic model of deep convolutional neural network, long short‐term memory, and deep neural network (DCNN‐LSTM‐DNN, DLD). This model utilizes DCNN to reduce frequency variation and adds a batch normalization (BN) layer after its convolutional layer to ensure the stability of data distribution, and then use LSTM to effectively solve the gradient vanishing problem. Finally, the fully connected structure of DNN is utilized to efficiently map the input features into a separable space, which is helpful for data classification. Therefore, leveraging the strengths of DCNN, LSTM, and DNN by combining them into a unified architecture can effectively improve speech recognition performance. Our model was tested on the open Chinese speech database THCHS‐30 released by the Center for Speech and Language Technology (CSLT) of Tsinghua University, and it was concluded that the DLD model with 3 layers of LSTM and 3 layers of DNN had the best performance, reaching 13.49% of words error rate (WER).

From Easy to Hard: Learning Language-Guided Curriculum for Visual Question Answering on Remote Sensing Data

Visual question answering (VQA) for remote sensing scene has great potential in intelligent human-computer interaction system. Although VQA in computer vision has been widely researched, VQA for remote sensing data (RSVQA) is still in its infancy. There are two characteristics that need to be specially considered for the RSVQA task. 1) No object annotations are available in RSVQA datasets, which makes it difficult for models to exploit informative region representation; 2) There are questions with clearly different difficulty levels for each image in the RSVQA task. Directly training a model with questions in a random order may confuse the model and limit the performance. To address these two problems, in this paper, a multi-level visual feature learning method is proposed to jointly extract language-guided holistic and regional image features. Besides, a self-paced curriculum learning (SPCL)-based VQA model is developed to train networks with samples in an easy-to-hard way. To be more specific, a language-guided SPCL method with a soft weighting strategy is explored in this work. The proposed model is evaluated on three public datasets, and extensive experimental results show that the proposed RSVQA framework can achieve promising performance.

Sensors and Artificial Intelligence Methods and Algorithms for Human-Computer Intelligent Interaction: A Systematic Mapping Study.

To equip computers with human communication skills and to enable natural interaction between the computer and a human, intelligent solutions are required based on artificial intelligence (AI) methods, algorithms, and sensor technology. This study aimed at identifying and analyzing the state-of-the-art AI methods and algorithms and sensors technology in existing human–computer intelligent interaction (HCII) research to explore trends in HCII research, categorize existing evidence, and identify potential directions for future research. We conduct a systematic mapping study of the HCII body of research. Four hundred fifty-four studies published in various journals and conferences between 2010 and 2021 were identified and analyzed. Studies in the HCII and IUI fields have primarily been focused on intelligent recognition of emotion, gestures, and facial expressions using sensors technology, such as the camera, EEG, Kinect, wearable sensors, eye tracker, gyroscope, and others. Researchers most often apply deep-learning and instance-based AI methods and algorithms. The support sector machine (SVM) is the most widely used algorithm for various kinds of recognition, primarily an emotion, facial expression, and gesture. The convolutional neural network (CNN) is the often-used deep-learning algorithm for emotion recognition, facial recognition, and gesture recognition solutions.

Design of Graphene Pressure Sensor Array and its Ethernet-Based Demodulation System

Graphene pressure sensors have some problems, such as difficulty in sampling, difficulty in real-time remote monitoring and the limited sensor nodes in the system. A graphene pressure sensor array and its Ethernet-based demodulation system is proposed in this work, and the automatic detection of low-pressure is realized. This demodulation system can construct a LAN with multiple pressure sensor nodes, and monitor any sensor node in the LAN through the remote host computer. The self-made graphene pressure sensor array has high sensitivity (0.303 kPa <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">−1</sup> in the range of less than 200 Pa and 0.038 kpa <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">−1</sup> in the range of 200-1000 pa) and stability in the low-pressure range (< 1 kPa), which meets the detection requirements of low pressure. The demodulation system has the characteristics of high-precision sampling, high-speed transmission and remote real-time dynamic display. This work has certain application prospects in wearable electronic devices, electronic skin, intelligent human-computer interaction and engineering applications.

Research on Key Methods of Visual Human-computer Interaction Comfort Quantifacation in Multidimensional Information Space

With the increase of information exchange types and the acceleration of information exchange speed, people not only pursue fast and efficient interaction effect, but also begin to pursue natural and harmonious human-computer interaction comfort experience physically and psychologically. Therefore, human perceived comfort has become an important index in the design of modern human-computer interaction system. However, since comfort is the subjective feeling of human body, it is difficult to realize quantitative measurement and evaluation in the design process, which brings great difficulties to computer intelligent human-computer interaction design and scientific evaluation. Vision is the most important sense of human beings. More than 80% of all kinds of information received by people from the outside world is obtained through vision. Vision is an important means of human-computer interaction in complex multi-dimensional information space, and it is also the most intuitive source and effect embodiment of human comfort perception. Through a large number of experiments, import the test data into the software Origin, and draw the change curve of visual comfort with reading time, which fit with the S-shaped curve as the reference. In this way, the relationship between visual comfort and reading time can be obtained. The mathematical model of the relationship between reading time and visual comfort is established, and experiments are carried out to verify the corresponding mathematical model relationship.

A Generative Adversarial Network Model Based on Intelligent Data Analytics for Music Emotion Recognition under IoT

The popularity of the Internet has brought the rapid development of artificial intelligence, affective computing, Internet of things (IoT), and other technologies. Particularly, the development of IoT provides more references for the realization of smart home. However, when people have achieved a certain amount of material satisfaction, they are more likely to want to communicate emotionally. Music contains a lot of emotion information. Music data is an important communication way between people and a better way to convey emotions. Therefore, it has become one of the most convenient and natural interactive ways expected by people in intelligent human-computer interaction. Traditional music emotion recognition methods have some demerits such as low recognition rate and time-consuming. So, we propose a generative adversarial network (GAN) model based on intelligent data analytics for music emotion recognition under IoT. Driven by the double-channel fusion strategy, the GAN can effectively extract the local and global features of the image or voice. Meanwhile, in order to increase the feature difference between the emotional voices, the feature data matrix of the Meyer frequency cepstrum coefficient of the music signals is transformed to improve the expression ability of the GAN. The experiment results show that the proposed model can effectively recognize the music emotion. Compared with other state-of-the-art approaches, the error recognition rate of proposed music music data recognition is greatly reduced. In terms of the accuracy, it exceeds 87% which is higher than that of other methods.

Design of Intelligent Human-Computer Interaction System for Hard of Hearing and Non-Disabled People

Since the hard of hearing cannot communicate effectively with the non-disabled, which may cause various inconveniences. As an essential member of a harmonious society, it is particularly urgent to solve their communication problems with non-disabled people. Effective communication between the hard of hearing and the non-disabled has become possible with the continuous development of artificial intelligence. In this paper, an intelligent human-computer interaction system is designed to solve communication inconvenience between the hard of hearing and the non-disabled. This system combines artificial intelligence with wearable devices and classifies gestures with BP neural network, effectively solving the communication problem between the hard of hearing and the non-disabled.

AI-Based Chatbot to Solve Modern-Day Enterprise Business Problems

Communicating with employees through forums and emails has become an increasingly popular way for many multinational companies to provide human resource services in real time. Today, employee chat service agents are often replaced by conversational software agents or chatbots. These systems are designed to communicate with human users through natural language, generally based on artificial intelligence (AI). Time and cost saving opportunities have led to the widespread deployment of AI-based chatbots. Chatbots are one of the most basic and popular examples of human-computer intelligent interaction (HCI). Designed to convincingly simulate the way humans behave as dialogue partners. In the proposed system, we propose a chat robot that can dynamically respond to employee human resource queries. The proposed HR system is based on the Microsoft Cognitive Services chatbot. This Microsoft Teams-based platform provides a broad foundation of intelligence and is trained based on various data sets provided by the organization's HR.

Human Motion Recognition Based on Multimodal Characteristics of Learning Quality in Football Scene

The task of human motion recognition based on video is widely concerned, and its research results have been widely used in intelligent human-computer interaction, virtual reality, intelligent monitoring, security, multimedia content analysis, etc. The purpose of this study is to explore the human action recognition in the football scene combined with learning quality related multimodal features. The method used in this study is to select BN-Inception as the underlying feature extraction network and use uncontrolled environment and real world to capture datasets UCFl01 and HMDB51, and pretraining is carried out on the ImageNet dataset. The spatial depth convolution network takes image frame as input, and the temporal depth convolution network takes stacked optical flow as input to carry out human action multimodal identification. In the results of multimodal feature fusion, the accuracy of UCFl01 dataset is generally high, all of which are over 80%, and the highest is 95.2%, while the accuracy of HMDB51 dataset is about 70%, and the lowest is only 56.3%. It can be concluded that the method of this study has higher accuracy and better effect in multimodal feature acquisition, and the accuracy of single-mode feature recognition is significantly lower than that of multimodal feature recognition. It provides an effective method for the multimodal feature of human motion recognition in the scene of football or sports.

The Influence of Key Facial Features on Recognition of Emotion in Cartoon Faces.

Cartoon faces are widely used in social media, animation production, and social robots because of their attractive ability to convey different emotional information. Despite their popular applications, the mechanisms of recognizing emotional expressions in cartoon faces are still unclear. Therefore, three experiments were conducted in this study to systematically explore a recognition process for emotional cartoon expressions (happy, sad, and neutral) and to examine the influence of key facial features (mouth, eyes, and eyebrows) on emotion recognition. Across the experiments, three presentation conditions were employed: (1) a full face; (2) individual feature only (with two other features concealed); and (3) one feature concealed with two other features presented. The cartoon face images used in this study were converted from a set of real faces acted by Chinese posers, and the observers were Chinese. The results show that happy cartoon expressions were recognized more accurately than neutral and sad expressions, which was consistent with the happiness recognition advantage revealed in real face studies. Compared with real facial expressions, sad cartoon expressions were perceived as sadder, and happy cartoon expressions were perceived as less happy, regardless of whether full-face or single facial features were viewed. For cartoon faces, the mouth was demonstrated to be a feature that is sufficient and necessary for the recognition of happiness, and the eyebrows were sufficient and necessary for the recognition of sadness. This study helps to clarify the perception mechanism underlying emotion recognition in cartoon faces and sheds some light on directions for future research on intelligent human-computer interactions.

Design of Three-Dimensional Gesture Recognition and Motion Tracking Human-Computer Intelligent Interaction System based on PAJ7620

With the increasing degree of informatization and intelligence in modern society, human-computer interaction technology has become one of the commonly used scientific and technological achievements. “Gesture” can be used as inputing image information, and make more intuitive and convenient for perception of user interface. To a certain extent, it can complete the intelligent interaction function between human and computer. The design is a human-computer intelligent interaction system for 3D gesture recognition and motion tracking based on gesture sensor PAJ7620. STM32 chip is used as the main control unit to recognize accurately, analyze and process different gestures collected by the gesture sensor, and control the OLED screen to display the gesture information. It can further drive the carrier to take corresponding actions and realize the intelligent human-computer interaction recognition of gestures. The experimental test results show that the recognition rate is lose to 100% when the distance is within 10cm.

Ontology modelling of intelligent HCI in aircraft cockpit

PurposeSemantic modelling is an essential prerequisite for designing the intelligent human–computer interaction in future aircraft cockpit. The purpose of this paper is to outline an ontology-based solution to this issue.Design/methodology/approachThe scenario elements are defined considering the cognitive behaviours, system functions, interaction behaviours and interaction situation. The knowledge model consists of a five-tuple array including concepts, relations, functions, axioms and instances. Using the theory of belief-desire-intention, the meta-model of cognitive behaviours is established. The meta-model of system functions is formed under the architecture of sub-functions. Supported by information flows, the meta-model of interaction behaviours is presented. Based on the socio-technical characteristics, the meta-model of interaction situation is proposed. The knowledge representation and reasoning process is visualized with the semantic web rule language (SWRL) on the Protégé platform. Finally, verification and evaluation are carried out to assess the rationality and quality of the ontology model. Application scenarios of the proposed modelling method are also illustrated.FindingsVerification results show that the knowledge reasoning based on SWRL rules can further enrich the knowledge base in terms of instance attributes and thereby improve the adaptability and learning ability of the ontology model in different simulations. Evaluation results show that the ontology model has a good quality with high cohesion and low coupling.Practical implicationsThe approach presented in this paper can be applied to model complex human–machine–environment systems, from a semantics-driven perspective, especially for designing future cockpits.Originality/valueDifferent from the traditional approaches, the method proposed in this paper tries to deal with the socio-technical modelling issues concerning multidimensional information semantics. Meanwhile, the constructed model has the ability of autonomous reasoning to adapt to complex situations.

RETRACTED ARTICLE: Art image simulation design of craft products based on virtual reality and human–computer interactive processing

At present, virtual reality technology is the most researched technology in various high-tech industries at home and abroad. Because virtual reality technology can produce three-dimensional animation that is very similar to reality, and with the help of the core technology of the technology simulation system, it provides consumers with an immersive information interaction experience, which greatly reduces the user’s learning of a certain new knowledge. Time and energy invested. One such virtual reality technology was used in the 2010 World Expo Online. Intelligent transportation is a combination of traditional road transportation infrastructure and modern high-tech information technology, and it is an important transportation mode for realizing future vehicle–road coordination. In terms of intelligent transportation, human–computer intelligent interaction technology determines the operating efficiency of the system and the corresponding service quality. In the process of actual use, the researchers developed LCD and audio decoder drivers for the system, and used the UDP protocol to establish a traffic information communication channel between the embedded system and the server, and through the Linux system as the basis. The principle of communication has added the function of human–computer intelligent interaction. This research clarified the basic principles of SMS and the design method of human–computer interaction system based on SMS technology. The researchers established the communication channel between the SMS module and the PC with the help of RS-232 technology, used the Visualc++ program development platform and multi-threaded operation technology to realize the function of sending and receiving messages in real time, and allowed the system to recognize the user's sending according to the set rules Instructions. Applying image recognition technology to actual production, constructing the time digital information image database, characteristic database and corresponding recognition database, this is also a preliminary attempt to apply image recognition technology in the field of stone carving art. Through the establishment of an automatic recognition system for tomb stone carvings in the Southern Song Dynasty, the high-tech image recognition technology is applied to the field of art, and the recognition accuracy of users in the process of recognizing historical pictures is improved, and the efficiency of recognizing historical damaged images is improved. The first is an image recognition zoom algorithm based on variational and wavelet transform. The Besov function is used to create a variational functional for the regularity of the historical image, and then the enlarged historical image is obtained by finding the minimum value of the variational functional, Convert the minimum value of this function to wavelet transform and use the program iteration method to solve; secondly, it is the historical image enlargement algorithm based on the curve and surface evolution, which uses the internal parameters of the system as the standard prior of the historical image Model, perform corresponding calculations on the model, and obtain the evolution equation of the gradient descent flow about the horizontal line of the image.

Facial expression recognition via ResNet-50

As one of the most important directions in the field of computer vision, facial emotion recognition plays an important role in people's daily work and life. Human emotion recognition based on facial expressions is of great significance in the application of intelligent human-computer interaction. However, in the current research on facial emotion recognition, there are some problems such as poor generalization ability of network model and low robustness of recognition system. In this content, we propose a method of feature extraction using the deep residual network ResNet-50, which combines convolutional neural network for facial emotion recognition. Through the experimental simulation of the specified data set, it can be proved that this model is superior to the current mainstream facial emotion recognition models in the performance of facial emotion detection.

MindLink-Eumpy: An Open-Source Python Toolbox for Multimodal Emotion Recognition.

Emotion recognition plays an important role in intelligent human–computer interaction, but the related research still faces the problems of low accuracy and subject dependence. In this paper, an open-source software toolbox called MindLink-Eumpy is developed to recognize emotions by integrating electroencephalogram (EEG) and facial expression information. MindLink-Eumpy first applies a series of tools to automatically obtain physiological data from subjects and then analyzes the obtained facial expression data and EEG data, respectively, and finally fuses the two different signals at a decision level. In the detection of facial expressions, the algorithm used by MindLink-Eumpy is a multitask convolutional neural network (CNN) based on transfer learning technique. In the detection of EEG, MindLink-Eumpy provides two algorithms, including a subject-dependent model based on support vector machine (SVM) and a subject-independent model based on long short-term memory network (LSTM). In the decision-level fusion, weight enumerator and AdaBoost technique are applied to combine the predictions of SVM and CNN. We conducted two offline experiments on the Database for Emotion Analysis Using Physiological Signals (DEAP) dataset and the Multimodal Database for Affect Recognition and Implicit Tagging (MAHNOB-HCI) dataset, respectively, and conducted an online experiment on 15 healthy subjects. The results show that multimodal methods outperform single-modal methods in both offline and online experiments. In the subject-dependent condition, the multimodal method achieved an accuracy of 71.00% in the valence dimension and an accuracy of 72.14% in the arousal dimension. In the subject-independent condition, the LSTM-based method achieved an accuracy of 78.56% in the valence dimension and an accuracy of 77.22% in the arousal dimension. The feasibility and efficiency of MindLink-Eumpy for emotion recognition is thus demonstrated.

A Two-Stage Spatiotemporal Attention Convolution Network for Continuous Dimensional Emotion Recognition From Facial Video

Continuous dimensional emotion recognition for facial video sequence is a crucial and challenging task in Affective Computing and Human-Computer Intelligent Interaction. The key of this task is to effectively extract and discriminate spatial-temporal features in a more fine-grained way. In this paper, a Two-Stage Spatiotemporal Attention Temporal Convolution Network (TS-SATCN) is designed for continuous dimensional emotion recognition of facial videos. The first stage generates an initial recognition result that is later fed into the second for correction. In each stage, the introduced spatiotemporal attention branch helps the network learn different attention levels and focuses on the informative spatial-temporal features adaptively. The network is trained by a proposed smooth loss function which can further improve the predictions' quality. Extensive experiments are performed on two datasets, RECOLA and AFEW-VA, which shows that the proposed method achieves significant improvement over state-of-the-art methods.

Self-Powered Intelligent Human-Machine Interaction for Handwriting Recognition.

Handwritten signatures widely exist in our daily lives. The main challenge of signal recognition on handwriting is in the development of approaches to obtain information effectively. External mechanical signals can be easily detected by triboelectric nanogenerators which can provide immediate opportunities for building new types of active sensors capable of recording handwritten signals. In this work, we report an intelligent human-machine interaction interface based on a triboelectric nanogenerator. Using the horizontal-vertical symmetrical electrode array, the handwritten triboelectric signal can be recorded without external energy supply. Combined with supervised machine learning methods, it can successfully recognize handwritten English letters, Chinese characters, and Arabic numerals. The principal component analysis algorithm preprocesses the triboelectric signal data to reduce the complexity of the neural network in the machine learning process. Further, it can realize the anticounterfeiting recognition of writing habits by controlling the samples input to the neural network. The results show that the intelligent human-computer interaction interface has broad application prospects in signature security and human-computer interaction.