Human-computer Interaction Applications Research Articles

TQU-HG dataset and comparative study for hand gesture recognition of RGB-based images using deep learning

Hand gesture recognition has great applications in human-computer interaction (HCI), human-robot interaction (HRI), and supporting the deaf and mute. To build a hand gesture recognition model using deep learning (DL) with high results then needs to be trained on many data and in many different conditions and contexts. In this paper, we publish the TQU-HG dataset of large RGB images with low resolution (640×480) pixels, low light conditions, and fast speed (16 fps). TQU-HG dataset includes 60,000 images collected from 20 people (10 male, 10 female) with 15 gestures of both left and right hands. A comparative study with two branches: i) based on Mediapipe TML and ii) Based on convolutional neural networks (CNNs) (you only look once (YOLO); YOLOv5, YOLOv6, YOLOv7, YOLOv8, YOLO-Nas, single shot multiBox detector (SSD) VGG16, residual network (ResNet)18, ResNext50, ResNet152, ResNext50, MobileNet V3 small, and MobileNet V3 large), the architecture and operation of CNNs models are also introduced in detail. We especially fine-tune the model and evaluate it on TQU-HG and HaGRID datasets. The quantitative results of the training and testing are presented (F1-score of YOLOv8, YOLO-Nas, MobileNet V3 small, ResNet50 is 98.99%, 98.98%, 99.27%, 99.36%, respectively on the TQU-HG dataset and is 99.21%, 99.37%, 99.36%, 86.4%, 98.3%, respectively on the HaGRID dataset). The computation time of YOLOv8 is 6.19 fps on the CPU and 18.28 fps on the GPU.

Interactive learning of natural sign language with radar

AbstractOver the past decade, there have been great advancements in radio frequency sensor technology for human–computer interaction applications, such as gesture recognition, and human activity recognition more broadly. While there is a significant amount of study on these topics, in most cases, experimental data are acquired in controlled settings by directing participants what motion to articulate. However, especially for communicative motions, such as sign language, such directed data sets do not accurately capture natural, in situ articulations. This results in a difference in the distribution of directed American Sign Language (ASL) versus natural ASL, which severely degrades natural sign language recognition in real‐world scenarios. To overcome these challenges and acquire more representative data for training deep models, the authors develop an interactive gaming environment, ChessSIGN, which records video and radar data of participants as they play the game without any external direction. The authors investigate various ways of generating synthetic samples from directed ASL data, but show that ultimately such data does not offer much improvement over just initialising using imagery from ImageNet. In contrast, an interactive learning paradigm is proposed by the authors in which model training is shown to improve as more and more natural ASL samples are acquired and augmented via synthetic samples generated from a physics‐aware generative adversarial network. The authors show that the proposed approach enables the recognition of natural ASL in a real‐world setting, achieving an accuracy of 69% for 29 ASL signs—a 60% improvement over conventional training with directed ASL data.

Ecofriendly and high-performance flexible pressure sensor derived from natural plant materials for intelligent audible and silent speech recognition

Flexible pressure sensors (FPSs) are widely used in human–computer interaction (HCI) and medical diagnoses, but their further development faces several challenges. Integrating artificial intelligence (AI) with FPSs is indispensable for advancing the cognitive capabilities of HCI systems, but the FPS must also be ecofriendly, cost-effective, and high-performance to meet the demands of environmental protection and sustainable development. Here, a biodegradable capacitive FPS is presented that is fabricated from natural plant materials (i.e., cornstarch and lignocellulose) while exhibiting remarkable attributes such as high sensitivity (2.427 kPa−1 at 0–2 kPa), rapid response/recovery time (20/24 ms), extremely low detection limit (∼10 Pa), outstanding stability (> 6000 fatigue tests), excellent biocompatibility (survival rate exceeding 99 % in cell viability assays), and prominent biodegradability (complete degradation within 24 h). Prepared sensors were integrated with optimized AI algorithms, embedded systems, and wireless transmission technologies to develop a dual-functional (i.e., audible and silent) speech recognition system that achieved recognition rates exceeding 95 %. The developed sensor is expected to play a crucial role in future HCI and medical assistance applications.

Efficient dual-frequency SSVEP brain-computer interface system exploiting interocular visual resource disparities

Here, we present a novel dual-frequency steady-state visual evoked potential (SSVEP) brain-computer interface (BCI) system with a unique human–computer interaction (HCI) feature that utilizes distinct visual resource disparities between the two eyes. By leveraging subtle differences in electroencephalography (EEG) responses due to binocular disparity, our system effectively distinguishes identical frequency targets. We conducted a two-phase validation: initial offline experiments confirmed EEG differences from interocular disparity, followed by global optimization to refine the SSVEP coding and spatial arrangement, allowing for 40 distinct targets within 20 frequency combinationes. A tailored binocular-task-related component analysis (bTRCA) algorithm was developed, achieving a maximum individual information transmission rate (ITR) of 338.32 bits/min offline and an average of 224.96 bits/min, with the online version reaching an average ITR of 234.13 bits/min. These results indicate that this system is a leading candidate for dual-frequency SSVEP BCIs, opening new prospects for HCI applications.

Facial Emotion Detection: Applications, Advancements, and Implications in Human-Computer Interaction

Abstract: The YOLO Version 8 algorithm, considered for its real-time object detection abilities, is implemented to demonstrate the face emotion detection. We integrate deep neural network analysis, accurate landmark detection, and face image preprocessing to achieve enhanced effectiveness and precision in real-time emotion recognition. This is made possible by YOLO V8's sophisticated architecture, which effectively processes visual input. Robust performance in various demographic and cultural contexts is guaranteed by extensive training on a variety of datasets. Comprehensive evaluations indicate that the system surpasses existing methodologies in terms of precision, efficiency, and adaptability. Anticipated advancements in emotion-aware technologies promise far-reaching implications, with potential applications spanning sentiment analysis, interactive computing, and mental well-being monitoring.

Athletes' Action Recognition and Posture Estimation Algorithm Based on Image Processing Technology

A posture estimation algorithm based on image processing technology involves analyzing images or video frames to determine the position and orientation of human bodies or specific body parts. This process typically begins with detecting key points or landmarks on the body, such as joints or anatomical features. This paper presents a novel approach for athletes' action recognition and posture estimation using image processing technology, specifically focusing on the Contour Mapping Multivariant Recognition and Estimation (CMMRE) algorithm. The primary objective is to develop a robust system capable of accurately identifying human actions and estimating postures from visual data, with potential applications in sports analysis, surveillance, and human-computer interaction. The CMMRE algorithm employs advanced techniques for contour mapping, feature extraction, and prediction, leveraging deep learning methodologies to analyze and interpret visual information effectively. Through a series of experiments and simulations, the algorithm's performance is evaluated, showcasing its ability to achieve high accuracy in action recognition tasks across various scenarios. The results highlight the algorithm's strengths in accurately predicting actions and estimating postures, while also identifying areas for improvement. The algorithm achieved an average accuracy of 95% in recognizing walking and running actions, and 80% in identifying jumping actions. The results highlight the algorithm's strengths in accurately predicting actions and estimating postures, while also identifying areas for improvement.

A Systematic Literature Review on Human Activity Recognition

Human Activity Recognition (HAR) plays a significant role in several fields by automatically identifying and monitoring human activities using advanced techniques. It enhances safety, improves healthcare services, optimizes fitness routines, and enables context-aware applications in various fields. HAR contributes to a more efficient and intelligent interaction between humans and technology. It has emerged as an essential research domain with applications in healthcare, smart environments, and human-computer interaction. This study aims to provide a comprehensive survey of the evolving landscape of HAR, including key methodologies, techniques, and trends in existing research. The study discusses various applications of HAR and their significance in modern smart environments. The survey also highlights different types of HAR and data collection techniques. Additionally, it explores various methods for analyzing the collected data and provides a comprehensive analysis of existing human activity classification datasets. It offers valuable insights into understanding the strengths and limitations of various HAR techniques. The study also discusses various challenges and future directions for HAR.

Effect of Positive-Negative Image Ratio on the Performance of Pedestrian Detection Model

Pedestrian detection holds significant importance in computer vision, finding applications in video surveillance, human-computer interaction, and autonomous vehicles. Surprisingly, there is a scarcity of research addressing the optimal ratio of positive to negative images for training detection models. This study endeavors to fill this research gap by exploring various detection models and determining the ideal ratio. Two distinct scenarios are investigated, each characterized by an equal total image count and an equivalent number of positive images sourced from CVC-14 night/visible, night/FIR, and INRIA databases. The study leverages the Histogram of Oriented Gradient, utilizing both Support Vector Machines and Medium Neural Networks to construct the detection models. Notably, the experiments reveal that the accuracy of the models remains relatively stable, even with an increase in the ratio of negative images. However, a noteworthy inverse relationship between sensitivity and specificity emerges as the ratio escalates. The findings, guided by the Youden Index, pinpoint the optimal training ratio for pedestrian detection models, falling within the range of 1:0.5 to 1:2In the CVC-14 nighttime database, the Youden index reached 100% when the model was trained with a 1:0.5 ratio using SVM, and a total of 4500 images were employed in the training process. On the other hand, in the INRIA dataset, the Youden index exhibited its highest value at 98.50%. This occurred when both SVM and a Medium neural network were utilized to train the model with a ratio of 1:2, utilizing a total of 3000 images for the training phase. It's worth highlighting that the processing time for SVM models lags behind that of Medium Neural Networks. This disparity arises from the heightened computational complexity inherent to medium-sized neural networks, making them computationally demanding compared to SVMs. This study contributes valuable insights into the nuanced relationship between image ratios and the performance of pedestrian detection models.

Sign Language Recognition Using Convolutional Neural Network

This project introduces a real-time sign language detection system powered by Convolutional Neural Networks (CNNs), designed to aid individuals with hearing impairments in communication. Leveraging OpenCV for video capture and hand detection, along with custom modules for hand tracking and image classification, the system seamlessly integrates deep learning methodologies. A pre-trained CNN model, trained on a comprehensive dataset of sign language gestures, forms the core of the system, ensuring accurate classification in real-time. By capturing video frames from a webcam, detecting hands, and processing them through the CNN model, the system provides immediate feedback by overlaying predicted gestures onto the video stream. Through its effective implementation, this project underscores the potential of CNNs in facilitating accessibility and inclusivity, while also paving the way for future enhancements and applications in assistive technology and human-computer interaction. Top of Form Keywords— Real-time, Sign Language Detection, Convolutional Neural Networks (CNNs), Communication, Hearing Impairments, Deep Learning.

Robust Orientation Estimation from MEMS Magnetic, Angular Rate, and Gravity (MARG) Modules for Human-Computer Interaction.

While the availability of low-cost micro electro-mechanical systems (MEMS) accelerometers, gyroscopes, and magnetometers initially seemed to promise the possibility of using them to easily track the position and orientation of virtually any object that they could be attached to, this promise has not yet been fulfilled. Navigation-grade accelerometers and gyroscopes have long been the basis for tracking ships and aircraft, but the signals from low-cost MEMS accelerometers and gyroscopes are still orders of magnitude poorer in quality (e.g., bias stability). Therefore, the applications of MEMS inertial measurement units (IMUs), containing tri-axial accelerometers and gyroscopes, are currently not as extensive as they were expected to be. Even the addition of MEMS tri-axial magnetometers, to conform magnetic, angular rate, and gravity (MARG) sensor modules, has not fully overcome the challenges involved in using these modules for long-term orientation estimation, which would be of great benefit for the tracking of human-computer hand-held controllers or tracking of Internet-Of-Things (IoT) devices. Here, we present an algorithm, GMVDμK (or simply GMVDK), that aims at taking full advantage of all the signals available from a MARG module to robustly estimate its orientation, while preventing damaging overcorrections, within the context of a human-computer interaction application. Through experimental comparison, we show that GMVDK is more robust to magnetic disturbances than three other MARG orientation estimation algorithms in representative trials.

CSINet: Channel–Spatial Fusion Networks for Asymmetric Facial Expression Recognition

Occlusion or posture change of the face in natural scenes has typical asymmetry; however, an asymmetric face plays a key part in the lack of information available for facial expression recognition. To solve the problem of low accuracy of asymmetric facial expression recognition, this paper proposes a fusion of channel global features and a spatial local information expression recognition network called the “Channel–Spatial Integration Network” (CSINet). First, to extract the underlying detail information and deepen the network, the attention residual module with a redundant information filtering function is designed, and the backbone feature-extraction network is constituted by module stacking. Second, considering the loss of information in the local key area of face occlusion, the channel–spatial fusion structure is constructed, and the channel features and spatial features are combined to enhance the accuracy of occluded facial recognition. Finally, before the full connection layer, more local spatial information is embedded into the global channel information to capture the relationship between different channel–spatial targets, which improves the accuracy of feature expression. Experimental results on the natural scene facial expression data sets RAF-DB and FERPlus show that the recognition accuracies of the modeling approach proposed in this paper are 89.67% and 90.83%, which are 13.24% and 11.52% higher than that of the baseline network ResNet50, respectively. Compared with the latest facial expression recognition methods such as CVT, PACVT, etc., the method in this paper obtains better evaluation results of masked facial expression recognition, which provides certain theoretical and technical references for daily facial emotion analysis and human–computer interaction applications.

Design and Optimization of Human-Computer Interaction System for Education Management Based on Artificial Intelligence

The continuous improvement and refinement of artificial intelligence (AI) technology has facilitated the broader application of human-computer interaction in the field of education management. The construction of an educational management human-computer interaction system based on AI technology can optimize and improve key parameters of educational management human-computer interaction scenarios, thereby creating a more comprehensive mobile learning (m-learning) application system. This paper is based on AI technology, analyzing gesture semantics and speech semantics, and combining fusion algorithms to construct an education management human interaction system. The performance changes of the system were compared with real experimental operations and the NOBOOK platform analysis. The results show that the education management human-computer interaction system constructed in this article can enhance the m-learning experience of participants. It ensures high recognition accuracy, leading to higher scores in all dimensions of indicator evaluation. Therefore, as one of the crucial forms of m-learning, the human-computer interaction system for education management based on AI can establish a foundation for the further enhancement and development of education management.

Research on Speech Emotion Recognition System Using Machine Learning

Speech Emotion Recognition (SER) remains a hot topic in the domain of affective computing, drawing considerable research interest. Its growing potential, advancements in algorithms, and real-world applications contribute to this interest. Human speech carries paralinguistic cues that can be quantitatively represented through features like pitch, intensity, and Mel-Frequency Cepstral Coefficients (MFCC). SER typically involves three main stages: data processing, feature extraction/selection, and classification based on emotional features present. These steps, tailored to the unique attributes of human speech, make machine learning methods a fitting choice for SER implementation. While recent studies in affective computing leverage various ML techniques for SER tasks, few delve into the techniques aiding these core steps. Moreover, the challenges within these stages and cutting-edge approaches addressing them often receive limited discussion or are overlooked in these works.This project introduces a pioneering Speech Emotion Recognition System leveraging BiLSTM Algorithm and Image Processing techniques, developed for implementation on the Raspberry Pi platform using Python. The system analyses recorded audio input, employing three LEDs for mood detection and interfacing with Raspberry Pi programming facilitated by an SD card. This abstract outlines an all-encompassing approach for real-time emotion recognition via audio analysis, catering to varied applications in emotional AI and human-computer interaction. KEYWORDS: Speech Emotion Recognition (SER), Machine Learning, Image Processing, Raspberry Pi.

Highly efficient recognition of similar objects based on ionic robotic tactile sensors

Tactile sensing provides robot the ability of object recognition, fine operation, natural interaction and so on. However, in the actual scenario, robotic tactile recognition of similar objects still faces difficult such as low efficiency and accuracy, resulting from lack of high-performance sensors and intelligent recognition algorithms. In this paper, a flexible sensor combining a pyramidal microstructure with a gradient conformal ionic gel coating was demonstrated, exhibiting excellent excellent signal-to-noise ratio (48 dB), low detection limit (1 Pa), high sensitivity (92.96 kPa-1), fast response time (55 ms), and outstanding stability over 15,000 compression-release cycles. Furthermore, a Pressure-Slip Dual-Branch Convolutional Neural Network (PSNet) architecture was proposed to separately extract hardness and texture features and perform feature fusion. In tactile experiments on different kinds of leaves, a recognition rate of 97.16% was achieved, and surpassed that of human hands recognition (72.5%). These researches showed the great potential in a broad application in bionic robots, intelligent prostheses, and precise human-computer interaction.

Designable high-performance TPU foam strain sensors towards human-machine interfaces

Flexible, breathable, and stable sensors were important in developing human–computer interaction systems and novel wearable devices. However, major challenges were still faced in two aspects, 1) high-performance strain sensors with a large sensing range, and 2) an interaction mode that is both convenient and efficient. Herein, we developed a thermoplastic polyurethane large-skeleton foam (TPU-LS foam) through a steam-induced method. The strain sensor comprised three layers of breathable foam, with CNTs/rGO TPU-LS foam as the inner layer and pure foam as the protective layer. PDMS/TiO2 was sprayed to create a UV-resistant and self-repairing coating, achieving a 66% melanin inhibition rate and a 91% repair rate. The sensor presented a wide work range over 100%, a high gauge factor of 637.8, and long-term durability. We trained machine learning frameworks to recognize different gestures and designed two types of human–computer interaction applications, allowing users to operate apps with gestures.

HCI Research and Innovation in China: A 10-Year Perspective

In the past years, human computer interaction (HCI) research and innovation have developed substantially, leading to a number of fruitful research topics. In this paper, we surveyed the HCI research and innovation in China from a 10-year perspective. We analyzed the popular research methodology and topics among Chinese researchers, including human modeling, user interface techniques, context awareness, user acceptance and performance, user experience design, human-AI interaction, HCI applications and social influences. We also conducted a bibliography analysis on the published papers in top-tier conferences and journals, which revealed a significant rising trend, and a generally broad distribution of research types. Moreover, we described typical applications and the industry influence of the research outcomes. We concluded with implications and reflections for HCI researchers across the world and shared the future research trends envisioned by Chinese researchers.

Two-Way Sign Language Translator for Visually Impaired People

Hearing disabilities affect people all over the world. Nearly 360 million people have a disabling hearing loss. In the recent years, there has been a rapid increase in the number of deaf and dumb victims due to birth defects, accidents and oral diseases. Since deaf and dumb people cannot communicate with a hearing person, they have to depend on some sort of visual communication. Sign Language is not understood by the majority of hearing people. The objective of our project is to form a bridge between the hearing-impaired community and the hearing community; to further commence a two-way communication. We propose a two-way sign language translator that converts Voice to sign language and vice versa in real time. We process the video frame by frame using the OpenCV Library in Python. Furthermore, we use background subtraction method called Gaussian Mixture-based Background/Foreground Segmentation Algorithm on each frame to subtract the background. Contours of this processed image are then passed into a Deep Neural Net (DNN) to classify the frame to the corresponding written language equivalent words. For conversion from sign language to voice, we use basic Natural Language Processing and gtts to accurately conserve the grammar in the sign language. The enrolment rate and literacy among the hearing-impaired children is far below the average for the population at large. This technology will then help the normal schools better integrate the hearing-impaired community thus making education more accessible and cheaper for them. Speech Emotion Recognition (SER) is a vital aspect of human-computer interaction and emotional intelligence applications. This SER that employs Mel- frequency cepstral coefficients (MFCC) as the basis for feature extraction, followed by a Convolutional Neural Network (CNN) with a customized architecture for emotion classification. The methodology 2 involves transforming audio samples into MFCC images, enabling the application of CNNs typically used in computer vision tasks. This fusion of traditional audio analysis with deep learning techniques harnesses the power of MFCC's spectral representation and CNN's spatial pattern recognition. By modifying the CNN architecture, the model can better discern emotional cues in the MFCC images, enhancing SER performance. This strategy offers SER a viable path forward, with the ability to identify complex emotional expressions. In order to increase the accuracy of emotion recognition, it tackles the requirement for feature-rich representations and sophisticated modelling techniques. KEYWORDS: OpenCV, DNN, SER, CNN, MFCC

Applying LSTM and GRU Methods to Recognize and Interpret Hand Gestures, Poses, and Face-Based Sign Language in Real Time

This research uses a real-time, human-computer interaction application to examine sign language recognition. This work develops a rule-based hand gesture approach for Indonesian sign language in order to interpret some words using a combination of hand movements, mimics, and poses. The main objective in this study is the recognition of sign language that is based on hand movements made in front of the body with one or two hands, movements which may involve switching between the left and right hand or may be combined with mimics and poses. To overcome this problem, a research framework is developed by coordinating hand gestures with poses and mimics to create features by using holistic MediaPipe. To train and test data in real time, the long short time memory (LSTM) and gated recurrent unit (GRU) approaches are used. The research findings presented in this paper show that hand gestures in real-time interactions are reliably recognized, and some words are interpreted with the high accuracy rates of 94% and 96% for the LSTM and GRU methods, respectively.

From Movements to Metrics: Evaluating Explainable AI Methods in Skeleton-Based Human Activity Recognition.

The advancement of deep learning in human activity recognition (HAR) using 3D skeleton data is critical for applications in healthcare, security, sports, and human-computer interaction. This paper tackles a well-known gap in the field, which is the lack of testing in the applicability and reliability of XAI evaluation metrics in the skeleton-based HAR domain. We have tested established XAI metrics, namely faithfulness and stability on Class Activation Mapping (CAM) and Gradient-weighted Class Activation Mapping (Grad-CAM) to address this problem. This study introduces a perturbation method that produces variations within the error tolerance of motion sensor tracking, ensuring the resultant skeletal data points remain within the plausible output range of human movement as captured by the tracking device. We used the NTU RGB+D 60 dataset and the EfficientGCN architecture for HAR model training and testing. The evaluation involved systematically perturbing the 3D skeleton data by applying controlled displacements at different magnitudes to assess the impact on XAI metric performance across multiple action classes. Our findings reveal that faithfulness may not consistently serve as a reliable metric across all classes for the EfficientGCN model, indicating its limited applicability in certain contexts. In contrast, stability proves to be a more robust metric, showing dependability across different perturbation magnitudes. Additionally, CAM and Grad-CAM yielded almost identical explanations, leading to closely similar metric outcomes. This suggests a need for the exploration of additional metrics and the application of more diverse XAI methods to broaden the understanding and effectiveness of XAI in skeleton-based HAR.

Review and Analysis of RGBT Single Object Tracking Methods: A Fusion Perspective

Visual tracking is a fundamental task in computer vision with significant practical applications in various domains, including surveillance, security, robotics, and human-computer interaction. However, it may face limitations in visible light data, such as low-light environments, occlusion, and camouflage, which can significantly reduce its accuracy. To cope with these challenges, researchers have explored the potential of combining the visible and infrared modalities to improve tracking performance. By leveraging the complementary strengths of visible and infrared data, RGB-infrared fusion tracking has emerged as a promising approach to address these limitations and improve tracking accuracy in challenging scenarios. In this paper, we present a review on RGB-infrared fusion tracking. Specifically, we categorize existing RGBT tracking methods into four categories based on their underlying architectures, feature representations, and fusion strategies, namely feature decoupling based method, feature selecting based method, collaborative graph tracking method, and traditional fusion method. Furthermore, we provide a critical analysis of their strengths, limitations, representative methods, and future research directions. To further demonstrate the advantages and disadvantages of these methods, we present a review of publicly available RGBT tracking datasets and analyze the main results on public datasets. Moreover,we discuss some limitations in RGBT tracking at present and provide some opportunities and future directions for RGBT visual tracking, such as dataset diversity, unsupervised and weakly supervised applications. In conclusion, our survey aims to serve as a useful resource for researchers and practitioners interested in the emerging field of RGBT tracking, and to promote further progress and innovation in this area.