Effective Human-computer Interaction Research Articles

Improving communication and promoting social inclusion for hearing-impaired users: Usability evaluation and design recommendations for assistive mobile applications.

This study examines the usability of communication-assistive applications for hearing-impaired users, with a focus on enhancing user experience and promoting social inclusion. Although such applications have been developed and evaluated previously, interface designs that consider the intimacy needs of hearing-impaired users remain under-explored. We performed a comprehensive usability evaluation employing a mixed-method approach, which involved hearing-impaired individuals as well as field experts. The findings revealed areas for improvement in the design, validated the feasibility of implementing these applications, and emphasized the importance of incorporating the unique needs and preferences of hearing-impaired users. Furthermore, this paper discusses the importance of introducing guidelines and evaluation scales for the "Design for Emotion and Life Knowledge" levels to facilitate smooth and effective human-computer interactions. Such measures will promote the development of intelligent assistive technologies that reflect the qualitative needs of people with disabilities and contribute to social rights for hearing-impaired users. With the growing demands of artificial-intelligence-powered assistive technologies, the inclusion of individuals with disabilities in the design and research process is anticipated to increase. In future, studies should be conducted to blend the culturally shared experiences and emotional bonds expressed by users (having mild-to-severe hearing impairment) with the design and development process of assistive devices or services.

Edge impulse-based convolutional neural network for Hand Posture Recognition

Hand Posture Recognition (HPR) plays a crucial role in enabling effective human-computer interaction, particularly for individuals with hearing disabilities. The study compares five models, including MobileNetV2 96x96 0.35, MobileNetV1 96x96 0.25, MobileNetV1 96x96 0.1, self-designed Network 1, and self-designed Network 2, based on the Sbastien Marcel Static Hand Posture Database. Evaluation metrics - infserencing time, peak RAM usage, flash usage, and accuracy - are used to analyze the performance. The experiment workflow for each model comprises five major steps. Firstly, a random selection of 120 images from the Sbastien Marcel Static Hand Posture Database is converted to JPG format. Then, the images are divided into 80% training data and 20% testing data. Subsequently, the original images are normalized, and features are extracted for further processing. Subsequently, the models are individually trained using the preprocessed data, optimizing their parameters. Finally, the trained models are evaluated using the testing data set to assess their performance in hand posture recognition. The results indicate that MobileNetV2 96x96 0.35 achieves the highest accuracy of 96.69% while consuming fewer hardware resources compared to other models. MobileNetV1 96x96 0.1 demonstrates the lowest inferencing time and peak RAM usage, making it suitable for real-time applications. Furthermore, self-designed Model 1 exhibits the lowest flash usage, making it a viable option for resource-constrained devices. This study provides valuable insights into the selection of CNN architectures for HPR, offering guidance for practitioners to choose models based on specific application requirements.

Gesture-Based Control of Multimedia Player Using Python and OpenCV

The increasing significance of computers in our daily lives, coupled with the rise of ubiquitous computing, has necessitated effective human-computer interaction. Hand gesture recognition systems have emerged as a real-time video-based solution for detecting and interpreting hand gestures, offering intelligent and natural human-computer interaction (HCI) methods. This project focuses on leveraging human hands as input devices for computer operation. Developed using Python and the OpenCV library, the program utilizes a computer webcam to capture and analyze hand shapes and patterns. The program provides real-time feedback by displaying recognized hand gestures on the live video stream. The ultimate outcome of this project is an application that enhances user experiences in contactless systems. The project recognizes and detects human hand motions using the Python computer language through a process flow that includes background subtraction, hand ROI segmentation, contour detection, and finger recognition. Techniques for processing images are used, including hand gesture detection, pattern recognition, thresholding, and contour detection. The processing of incoming photos and the creation of related keystrokes are made possible by OpenCV, a rich set of image processing tools

Optimizing Machine Learning Models in Human Activity Recognition with the WISDM Dataset

Human Activity Recognition (HAR) is a technology with significant potential, utilizing data from various devices like smartphones and cameras. It finds applications in daily activities such as driving, cleaning, and gaming, involving fundamental movements like standing, sitting, jogging, and typing. Accurate identification of these actions is crucial for effective human-computer interaction systems. This work incorporates a HAR module to extract valuable insights from data signals. It employs Machine Learning (ML) models to automatically detect human activities using raw data from Internet of Things wearable sensors, including innovative combinations like Adagrad and ELU within the MLP algorithm. The ML models' performance is evaluated using statistical metrics such as accuracy, precision, recall, and f1-score, and comparisons are made with existing models.

DGU-HAU: A Dataset for 3D Human Action Analysis on Utterances

Constructing diverse and complex multi-modal datasets is crucial for advancing human action analysis research, providing ground truth annotations for training deep learning networks, and enabling the development of robust models across real-world scenarios. Generating natural and contextually appropriate nonverbal gestures is essential for enhancing immersive and effective human–computer interactions in various applications. These applications include video games, embodied virtual assistants, and conversations within a metaverse. However, existing speech-related human datasets are focused on style transfer, so they have limitations that make them unsuitable for 3D human action analysis studies, such as human action recognition and generation. Therefore, we introduce a novel multi-modal dataset, DGU-HAU, a dataset for 3D human action on utterances that commonly occurs during daily life. We validate the dataset using a human action generation model, Action2Motion (A2M), a state-of-the-art 3D human action generation model.

Research on Intelligent Dialogue Systems

Intelligent dialogue systems, as a subfield of artificial intelligence, have very important research significance and application value. Todays AI dialogue systems are still in a relatively early stage, but they are developing very rapidly. In recent years, intelligent dialogue systems have been applied in many fields, such as intelligent customer service in online transactions, intelligent voice assistants in smartphones, and virtual chatbots. This paper introduces the background of intelligent dialogue systems and the current research status of key technologies and discusses some challenges in this field and some recent research to improve the system. Most of the current intelligent dialogue systems can perform effective human-computer interaction and respond accordingly. But for the next generation of intelligent dialogue system, more human characteristics are needed so that it can better understand and express human language, have its own personality, and maintain the consistency and logic of dialogue.

Emotion recognition framework using multiple modalities for an effective human–computer interaction

Emotion recognition framework using multiple modalities for an effective human–computer interaction

Human-Computer Interaction: Enhancing User Experience in Interactive Systems

Enhancing user experience (UX) in interactive systems requires effective human-computer interaction (HCI). The relationship between people and computers has grown in significance as technology progresses, having an impact on many areas of our life. The main ideas and tactics used in HCI to enhance the user experience in interactive systems are examined in this abstract. Understanding the capabilities and constraints of both humans and computers forms the basis of HCI. HCI researchers and designers may develop interactive systems that complement users' mental models and cognitive processes by researching human behavior, cognition, and psychology. Additionally, taking into account the features of the computer system, such as its responsiveness, processing capacity, and interface design, enables the development of systems that are more user-friendly and effective. As a result, the discipline of human-computer interaction works to improve the user experience in interactive systems. Researchers and designers in the field of human-computer interaction (HCI) can produce interactive systems that are simple, effective, entertaining, and satisfying for users by comprehending human capabilities, applying user-centered design principles, utilizing interactive techniques and interfaces, integrating multimodal interfaces, and embracing emerging technologies. The continued development of HCI will continue to influence and enhance how people use computers, enhancing user experiences and creating new opportunities for interactive systems in the future.

MEMS based eating and drinking gesture spotting using machine learning techniques

An effective human-computer interaction for collaborative, collective and brilliant computing is provided by the Gesture recognition. Our study uses a single 3-axis accelerometer and 3-axis gyroscope for acquiring the data. Dynamic Time Warping (DTW), time and frequency domain and gesture discrepancy features are extracted from the raw data and these features are subjected to supervised machine learning algorithms such as KNN, SVM, RF and ANN. The system is evaluated depending on the data base of more than 10,000 traces obtained from five subsets. Through this study, we intend to identify eating and drinking gestures and more broadly short activities on real time environment. Many approaches for gesture recognition in specific settings have already been explored and studied using the combined interaction of several sensors. Despite these powerful hypotheses, gesture interpretation is still fragile and often depends on the individual’s positioning relative to the cameras not with the sensors.

Hybrid Approach for Facial Expression Recognition Using Convolutional Neural Networks and SVM

Facial expression recognition is very useful for effective human–computer interaction, robot interfaces, and emotion-aware smart agent systems. This paper presents a new framework for facial expression recognition by using a hybrid model: a combination of convolutional neural networks (CNNs) and a support vector machine (SVM) classifier using dynamic facial expression data. In order to extract facial motion characteristics, dense facial motion flows and geometry landmark flows of facial expression sequences were used as inputs to the CNN and SVM classifier, respectively. CNN architectures for facial expression recognition from dense facial motion flows were proposed. The optimal weighting combination of the hybrid classifiers provides better facial expression recognition results than individual classifiers. The system has successfully classified seven facial expressions signalling anger, contempt, disgust, fear, happiness, sadness and surprise classes for the CK+ database, and facial expressions of anger, disgust, fear, happiness, sadness and surprise for the BU4D database. The recognition performance of the proposed system is 99.69% for the CK+ database and 94.69% for the BU4D database. The proposed method shows state-of-the-art results for the CK+ database and is proven to be effective for the BU4D database when compared with the previous schemes.

Exemplars-Guided Empathetic Response Generation Controlled by the Elements of Human Communication

Empathy is fundamental to humans amongst other animals. It is key to strong social cohesion, cornerstone of health and success of societies. Thus, empathy could be an important component of effective human-computer interactions through conversations. This has motivated a whole sub-field of research focused on empathetic response generation. The majority of existing methods for empathetic response generation rely on the emotion of the context to generate empathetic responses. However, empathy is much more than generating responses with an appropriate emotion. It also often entails subtle expressions of understanding and personal resonance with the situation of the other interlocutor. Unfortunately, such qualities are difficult to quantify, and the datasets lack the relevant annotations. To address this issue, in this paper we propose an approach that relies on exemplars to cue the generative model on fine stylistic properties that signal empathy to the interlocutor. To this end, we employ dense passage retrieval to extract relevant exemplary responses from the training set. Three elements of human communication—emotional presence, interpretation, and exploration—and sentiment are additionally introduced using synthetic labels to guide the generation towards empathy. The human evaluation is also extended by these elements of human communication. We empirically show that these approaches yield significant improvements in empathetic response quality in terms of both automated and human-evaluated metrics. The implementation is available at https://github.com/declare-lab/exemplary-empathy.

An Intelligent Hybrid–Integrated System Using Speech Recognition and a 3D Display for Early Childhood Education

In the past few years, people’s attitudes toward early childhood education (PAUD) have undergone a complete transformation. Personalized and intelligent communication methods are highly praised, which also promotes the further focus on timely and effective human–computer interaction. Since traditional English learning that relies on parents consumes more time and energy and is prone to errors and omissions, this paper proposes a system based on a convolution neural network (CNN) and automatic speech recognition (ASR) to achieve an integrated process of object recognition, intelligent speech interaction, and synchronization of learning records in children’s education. Compared with platforms described in the literature, not only does it shoot objects in the real-life environment to obtain English words, their pronunciation, and example sentences corresponding to them, but also it combines the technique of a three-dimensional display to help children learn abstract words. At the same time, the cloud database summarizes and tracks the learning progress by a horizontal comparison, which makes it convenient for parents to figure out the situation. The performance evaluation of image and speech recognition demonstrates that the overall accuracy remains above 96%. Through comprehensive experiments in different scenarios, we prove that the platform is suitable for children as an auxiliary method and cultivates their interest in learning English.

Transfer Learning-Based Muscle Activity Decoding Scheme by Low-frequency sEMG for Wearable Low-cost Application

The surface electromyogram (sEMG) contains a wealth of motion information, which can reflect user's muscle motion intentions. The decoding based on sEMG has been widely used to provide a safe and effective human-computer interaction (HCI) method for neural prosthesis and exoskeleton robot control. The motor intention decoding based on low sampling frequency sEMG may promote the application of wearable low-cost EMG sensors in HCI. Therefore, a motor intention decoding scheme suitable for low frequency EMG signal is proposed in this paper, that is, transfer learning based on Alexnet. Moreover, the effects of different feature extraction methods and data augmentation with Gaussian white noise are fully analyzed. The proposed algorithm is evaluated with the NinaPro database 5. The highest accuracy can reach 70.4%±4.36% in 53 gestures identification of 10 subjects. Some classical machine learning algorithms such as support vector machine (SVM), linear discriminant analysis (LDA) and K Nearest Neighbor (KNN) are chosen to make comparison, where the SVM with Gaussian kernel function reaches to the maximum accuracy of 67.98%±4.56%. Two-way variance results show significant differences between each other. The experiment results show that the transfer learning is effective for decoding low-frequency sEMG for a large number of gestures.

Household smart cat litter box

With the change and development of the times, more and more young people live alone in the cities, and with the pressure of work, they are often lack of company. Cats are docile in nature, like keeping neat and tidy, and don’t need too much time with their owners, so keeping pet cats is more convenient for most of people. Therefore, pet cats are very popular among young people, more and more young people start to keep pet cats as companions. In addition, with the improvement of people’s living standards, the average family can also afford to raise a cat. Whether as a companion to a child or a member of the family, pet cats are also very popular in ordinary families.According to the surveys, most families now have some pets. But obviously it’s a problem for these families to deal with the excrement of the cats. Because of the heavy smell, every time shoveling the feces will cause a lot of trouble to people. What’s worse, the smell of feces will evaporate in the air, causing air pollution. In order to solve the above problems, we designed a smart litter box that can separate and completely seal the cat feces. It solves the problem of the difficulty in handling cat feces and the heavy smell of cat feces, and can maximize the utilization of household space. Through the fully automated function and effective human-computer interaction function of this litter box, people can greatly save the time and energy spent in processing cat feces.

A Hybrid Technique using CNN LSTM for Speech Emotion Recognition

Automatic speech emotion recognition is a very necessary activity for effective human-computer interaction. This paper is motivated by using spectrograms as inputs to the hybrid deep convolutional LSTM for speech emotion recognition. In this study, we trained our proposed model using four convolutional layers for high-level feature extraction from input spectrograms, LSTM layer for accumulating long-term dependencies and finally two dense layers. Experimental results on the SAVEE database shows promising performance. Our proposed model is highly capable as it obtained an accuracy of 94.26%.

An Efficient Human Computer Interaction through Hand Gesture Using Deep Convolutional Neural Network

This paper focuses on the achievement of effective human–computer interaction using only webcam by continuous locating or tracking and recognizing the hand region. We detected the region of interest (ROI) in the captured image range and classify hand gestures for specific tasks. Firstly, background subtraction is used based on the main frame captured by webcam, and some preprocessing are done, and then YCrCb skin segmentation is used on RGB subtracted image. The ROI is detected using Haar cascade classifier for hand palm detection. Next, kernelized correlation filters tracking algorithm is used to avoid noise or background influences for tracking the ROI, and the median-flow tracking algorithm is used for depth tracking. The ROI is converted to a binary channel (black and white), resized to 54 × 54. Then gesture recognition is done using a 2D convolutional neural network (CNN) by entering the preprocessed ROI on the architecture. Two predictions are made based on skin segmented frame and image dilated frame, and gesture is recognized from the maximum value of those two predictions. The tracking and recognition process is continued until the ROI is presented on the frames. Finally, after validation, the proposed system has successfully obtained a recognition rate of 98.44%, which is usable for the practical and real-time application.

Gravitational search algorithm based optimized deep learning model with diverse set of features for facial expression recognition

Facial expression recognition (FER) is an essential part of effective human–computer interaction and serves as a helpful medium for children and patients who have problems with communication. However, most of the previous studies focus on building a FER model based on supervised and unsupervised approaches. This paper is focused on a semi-supervised deep belief network (DBN) approach to predict the facial expressions from the CK+, Oulu CASIA, MMI, and JAFFE datasets. To achieve accurate classification of the facial expressions, a gravitational search algorithm (GSA) is applied to optimize some parameters in the DBN network. The Histogram oriented gradients (HOG) and 2D-Discrete Wavelet Transform (2D-DWT) are used for feature extraction from the lip, cheek, brow, eye, and furrow patches. The unwanted information present in the image is eliminated using a feature selection approach. The feature extraction is done by the Kernel-principal component analysis to obtain higher-order correlations between input variables and detect non-linear components. The HOG features extracted from the lip patch provides the best performance for accurate facial expression classification. Finally, a comparative analysis to compare the proposed model with different machine learning techniques based on the evaluation criteria. The results demonstrate that DBN-GSA based classifier is more accurate than the rest of the classifiers.

Research trends in Hand Gesture Recognition techniques

Over years’ key board and mouse are among the most effective input devices to the computers. Not much improvement has happened for other modes of inputs like voice and gestures. These kinds of inputs explore the limitations caused by the key board and mouse, which needs actual physical contact. With gestures and voice the physical proximity can be little relaxed and can be used by specially-abled people. Exploring effective Human Computer Interactions (HCI) has always been of great interest for development of new techniques and methods. Gestures are considered as more easy, natural, and intuitive method for communicating with systems. Hand recognition and analysis forms the core of these systems. This paper provides comparative study on state of art technologies that work in this area. The different representation, detection and recognition techniques are discussed. The strength and challenges of different techniques are highlighted. The main aim of this research is to summarize the research progress done till date with different technologies for gesture recognition algorithms, identify areas which need further research.

Interview with Ben Shneiderman

Ben Shneiderman (http://www.cs.umd.edu/~ben) is a Distinguished University Professor in the Department of Computer Science, Founding Director (1983--2000) of the Human-Computer Interaction Laboratory (http://hcil.umd.edu), and a Member of the UM Institute for Advanced Computer Studies (UMIACS) at the University of Maryland. He is a Fellow of the AAAS, ACM, IEEE, and NAI, and a Member of the National Academy of Engineering, in recognition of his pioneering contributions to human-computer interaction and information visualization. His widely-used contributions include the clickable highlighted web-links, high-precision touchscreen keyboards for mobile devices, and tagging for photos. Shneiderman's information visualization innovations include dynamic query sliders for Spotfire, development of treemaps for viewing hierarchical data, novel network visualizations for NodeXL, and event sequence analysis for electronic health records. Ben is the co-author with Catherine Plaisant of Designing the User Interface: Strategies for Effective Human-Computer Interaction (6th ed., 2016). With Stu Card and Jock Mackinlay, he co-authored Readings in Information Visualization: Using Vision to Think (1999). His book Leonardo's Laptop (MIT Press) won the IEEE book award for Distinguished Literary Contribution. He co-authored, Analyzing Social Media Networks with NodeXL (Morgan Kaufmann) with Derek Hansen and Marc Smith. The New ABCs of Research: Achieving Breakthrough Collaborations (Oxford, April 2016) has an accompanying short book Twin-Win Research: Breakthrough Theories and Validated Solutions for Societal Benefit: Second Edition (2019).

EEG Based Emotion Recognition by Combining Functional Connectivity Network and Local Activations

Spectral power analysis plays a predominant role in electroencephalogram-based emotional recognition. It can reflect activity differences among multiple brain regions. In addition to activation difference, different emotions also involve different large-scale network during related information processing. In this paper, both information propagation patterns and activation difference in the brain were fused to improve the performance of emotional recognition. We constructed emotion-related brain networks with phase locking value and adopted a multiple feature fusion approach to combine the compensative activation and connection information for emotion recognition. Recognition results on three public emotional databases demonstrated that the combined features are superior to either single feature based on power distribution or network character. Furthermore, the conducted feature fusion analysis revealed the common characters between activation and connection patterns involved in the positive, neutral, and negative emotions for information processing. The proposed feasible combination of both information propagation patterns and activation difference in the brain is meaningful for developing the effective human-computer interaction systems by adapting to human emotions in the real world applications.