Aspects Of Human-computer Interaction Research Articles

Real-time Speech Emotion Recognition using Machine Learning

This project is centred around examining spoken language emotion, a critical aspect of human-computer interaction and artificial intelligence. Leveraging advanced techniques such as Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) models, we aim to decipher and understand the nuanced expressions of emotions conveyed through speech. The investigation draws from diverse datasets, including RAVDESS, SAVEE, TESS, and CREAM-D, each carefully chosen to encapsulate a broad spectrum of emotions and real-world speech scenarios. The project unfolds by elucidating the methodologies employed in dataset selection, the prepossessing of raw audio data, and the intricacies of utilizing CNN and LSTM techniques for speech emotion analysis. The main objective is to create a robust and reliable outcome model capable of accurately classifying and interpreting emotions in speech across various contexts. Practical applications of this Research extend to fields such as sentiment analysis and potential contributions to mental health monitoring. Through this project, we contribute valuable insights to the evolving landscape of speech emotion analysis, addressing the inherent challenges and exploring opportunities for enhancing human-computer interaction and emotional intelligence in artificial systems.

User-Centered Evaluation Framework to Support the Interaction Design for Augmented Reality Applications

User-Centered Evaluation Framework to Support the Interaction Design for Augmented Reality Applications

Using transformers for multimodal emotion recognition: Taxonomies and state of the art review

Emotion recognition is an aspect of human-computer interaction, affective computing, and social robotics. Conventional unimodal approaches for emotion recognition, depending on single data sources such as facial expressions or speech signals often fall short in capturing the complexity and context-dependent nature of emotions. Multimodal Emotion Recognition (MER), which integrates information from multiple modalities, has emerged as a promising solution to overcome these limitations. In recent years, Transformers-based approaches have gathered significant attention in the fields of natural language processing and computer vision, highlighting their ability to capture long-range dependencies and semantic representations. These models have rapidly achieved the MER state-of-the-art. However, current survey papers that cover MER lack a specific focus on Transformer-based techniques. To bridge this research gap, this review paper provides a comprehensive investigation of Transformers-based approaches for MER. It explores various Transformer architectures and proposes several scenarios for using Transformers at different stages of MER process. In addition, it examines datasets suitable for MER, discusses fusion mechanisms, and introduces novel taxonomies in both MER and Transformer technologies. The review also addresses challenges and future research directions. Through this review, we aim to provide researchers with an inclusive understanding of the current state-of-the-art in Transformers-based approaches for MER, paving the way for further advancements in this rapidly developing field.

Two-Way Sign Language Translator for Visually Impaired People

Hearing disabilities affect people all over the world. Nearly 360 million people have a disabling hearing loss. In the recent years, there has been a rapid increase in the number of deaf and dumb victims due to birth defects, accidents and oral diseases. Since deaf and dumb people cannot communicate with a hearing person, they have to depend on some sort of visual communication. Sign Language is not understood by the majority of hearing people. The objective of our project is to form a bridge between the hearing-impaired community and the hearing community; to further commence a two-way communication. We propose a two-way sign language translator that converts Voice to sign language and vice versa in real time. We process the video frame by frame using the OpenCV Library in Python. Furthermore, we use background subtraction method called Gaussian Mixture-based Background/Foreground Segmentation Algorithm on each frame to subtract the background. Contours of this processed image are then passed into a Deep Neural Net (DNN) to classify the frame to the corresponding written language equivalent words. For conversion from sign language to voice, we use basic Natural Language Processing and gtts to accurately conserve the grammar in the sign language. The enrolment rate and literacy among the hearing-impaired children is far below the average for the population at large. This technology will then help the normal schools better integrate the hearing-impaired community thus making education more accessible and cheaper for them. Speech Emotion Recognition (SER) is a vital aspect of human-computer interaction and emotional intelligence applications. This SER that employs Mel- frequency cepstral coefficients (MFCC) as the basis for feature extraction, followed by a Convolutional Neural Network (CNN) with a customized architecture for emotion classification. The methodology 2 involves transforming audio samples into MFCC images, enabling the application of CNNs typically used in computer vision tasks. This fusion of traditional audio analysis with deep learning techniques harnesses the power of MFCC's spectral representation and CNN's spatial pattern recognition. By modifying the CNN architecture, the model can better discern emotional cues in the MFCC images, enhancing SER performance. This strategy offers SER a viable path forward, with the ability to identify complex emotional expressions. In order to increase the accuracy of emotion recognition, it tackles the requirement for feature-rich representations and sophisticated modelling techniques. KEYWORDS: OpenCV, DNN, SER, CNN, MFCC

Application Programming Based on Human-Computer Interaction and Software Modularity

In the 21st century, cutting-edge information technologies such as artificial intelligence, blockchain and big data have profoundly affected the way human society is organised, the form of interaction and even the social structure. With the miniaturisation, mobility, intelligence and embedding of machines, the relationship between humans and machines has undergone fundamental changes. Human-computer interaction has gradually become one of the most important forms of interaction in human production and life, and human daily life centred on human-computer interaction has also undergone great changes. Various applications have been introduced into the lives of human beings, and at present, it would be difficult for human beings to carry out their daily lives without these applications as a vehicle for the development of information technology. However, complicated applications do not really enhance the usage experience in many cases, redundant information also ignores the real needs of users in this paper, which will focus on human-computer interaction, from the four aspects of human-computer interaction (HCI) HCI modes, HCI interfaces, HCI devices, HCI modalities, HCI patterns to analysethen borrow the concept of software modularity to optimise the current applications, and put forward some suggestions for improvement. This will make the application more suitable for the user's experience and reduce the burden of programme managers.

User perception as a factor for improving Trustworthiness in e-commerce systems

The User Interface (UI) is the first artefact that the user interacts with while developing a sense of trust that motivates him to use software applications more effectively. A badly designed UI can deceive users and bring the system into disrepute. Trustworthiness in UI is mandatory, as a poorly implemented UI can lead to the user misusing the system and jeopardizing the expected result. Trust in computational systems involves not only technical aspects, such as computational infrastructure, storage space, and service composition, but also aspects of Human-Computer Interaction (HCI). While technological aspects have received considerable attention, there are few research on human-computer interaction in terms of trust. This paper describes a way for assessing a system’s trustworthiness based on the user’s perception. The approach relies on a quality model to aggregate interface quality criteria in order to get a trustworthiness score. Three sets of experiments involving more than 300 individuals were carried out to validate the suggested methodology. A comparison was made between the trustworthiness score obtained through the methodology and the answers to open questions obtained through the users’ questionnaires. The results were consistent, and statistical analysis corroborated the positive assessment. Based on these results, examples of improvements were developed to highlight the usefulness of the approach for developing more trustworthiness interfaces.

Interakcija između čoveka i računara - ključni aspekti i evolucija

Introduction/purpose: The research aimed to analyze the key aspects of human-computer interaction, studying the evolution of this field from its beginnings to contemporary trends. Focusing on different aspects of this interaction, it studied how methods, technologies and principles have evolved over time and shaped our experiences in the digital environment. Methods: The research methodology was based on the analysis of relevant literature and research papers devoted to human-computer interaction. Systematic review and evaluation of sources identified the key information on the evolution of user interaction with computers. Through evaluation of available studies, changes in interface design methods, interaction technologies and understanding of user needs over time were identified. Results: Based on data synthesis, the key aspects of human-computer interaction were identified. An analysis of the literature revealed changes in interface design methods, interaction technologies, and understanding of user needs over time. Specific results included information on the evolution of interaction aspects, including improvements in efficiency, intuitiveness and personalization of experiences. Conclusion: The key conclusion of the research is that innovations and evolution of technology have significantly influenced the way users interact with computer systems. Improvements in interface design, recognition technologies, and personalization of experiences are key factors in creating better user experiences and optimizing human-computer interaction. The research points to the continuous development and changes in the field of human-computer interaction, emphasizing the need for further research and innovation in this area.

Influence Of Information Technology On Youth

Information technology (IT) is a set of methods and tools used to collect, store, process, and disseminate information. This is a general concept that describes various mechanisms, devices, algorithms, and data processing methods based on computer technology. At present, human life is increasingly dependent on IT technology in different ways. Every day, people go to social networks, check the news, chat with friends, and all this happens with the help of cellphones, which are representatives of modern IT technologies. At work, individuals most often perform their functions using computers and other technologies and devices (Lareki et al., 2017). Due to their use, society can perform various types of activities. From the point of view of development, modern technologies greatly simplify life in many ways and save time that can be used for comprehensive development and self-improvement. Information technologies are developing at an unprecedented pace, and they have conquered almost all spheres of human life. In this regard, a number of psychological problems have arisen associated with the development of personality, requiring theoretical understanding and integration of various areas of psychology. One of the issues is linked with the negative aspects of human-computer interaction and the resulting psychological dependencies, computer anxiety, and self-efficacy (Lareki et al., 2017). Nevertheless, there are also positive aspects of the influence of information and communication technologies on human mental activity. At the initial stage of using computers, psychological problems were associated with acquiring new technologies by a person. However, later on, scientists drew attention to emerging problems when using advanced technologies. In particular, this is a danger of withdrawal into cyberspace as a result of excessive enthusiasm for IT. Scientists have also described the positive aspects of the influence of information and communication technologies on personality development (Lareki et al., 2017). For example, by using IT technology, people can expand their knowledge dramatically and evolve their problem-solving skills.

Prospects of soft biopotential interfaces for wearable human-machine interactive devices and applications

Human interaction with machines can be made easy, comfortable, and accessible by introducing user-friendly interfaces. In the case of wearable devices, their sensors and other interfacing elements are very well within the proximity of users. Since biopotential signals can be accessed from the surface of the human skin, users can have seamless interaction with wearable human-computer interactive devices. Rigid interfaces can hinder the user experience, and therefore, the need for soft biopotential interfaces is important. Imperceptible and unobtrusive soft biopotential interfaces will drastically enhance many aspects of human-computer interaction. This paper reviews the use of soft, flexible, and stretchable biopotential interfaces in wearable human-machine interactive devices. Additionally, attention is brought to the scope of other possible applications of soft biopotential interfaces in wearable devices.

A multi-head residual connection GCN for EEG emotion recognition

Electroencephalography (EEG) emotion recognition is a crucial aspect of human-computer interaction. However, conventional neural networks have limitations in extracting profound EEG emotional features. This paper introduces a novel multi-head residual graph convolutional neural network (MRGCN) model that incorporates complex brain networks and graph convolution networks. The decomposition of multi-band differential entropy (DE) features exposes the temporal intricacy of emotion-linked brain activity, and the combination of short and long-distance brain networks can explore complex topological characteristics. Moreover, the residual-based architecture not only enhances performance but also augments classification stability across subjects. The visualization of brain network connectivity offers a practical technique for investigating emotional regulation mechanisms. The MRGCN model exhibits average classification accuracies of 95.8% and 98.9% for the DEAP and SEED datasets, respectively, highlighting its excellent performance and robustness.

Multi-modal speech emotion detection using optimised deep neural network classifier

ABSTRACT Speech emotion recognition (SER) has received significant attention recently and seems to be a critical aspect of human-computer interaction. The performance of the recent methods is not at the expected level there are many different strategies have been developed for SER. In the present situation, communication, non-verbal vocalisation, with a vocal sound plays an essential role in emotional expression. The multimodal database is considered for which the hybridised model of the audio-visual dependent emotion identification model is proposed. In this research, the optimised deep NN is used to recognise emotions from multimodal input data. For better performance in emotion recognition, the learner memorising optimisation fine-tunes the hyperparameters involved in the deep NN network classifier. The hybrid texture feature descriptor is proposed to improve classification outcomes from audio-video signals as well as accuracy. The learner memorising optimisation is used in the deep NN classifier to obtain the ideal parameters involved in exploring space with high accuracy. The metric values of the learner memorising optimisation-based deep NN have been evaluated using the Enter face’05 database and the proposed method obtained 97.490% accuracy, 98% sensitivity, and 97.490% specificity for the K-fold value 10% and 96.928%, 98.80%, and 96.928% for 90% training data.

Contactless Real-Time Eye Gaze-Mapping System Based on Simple Siamese Networks

Human–computer interaction (HCI) is a multidisciplinary field that investigates the interactions between humans and computer systems. HCI has facilitated the development of various digital technologies that aim to deliver optimal user experiences. Gaze recognition is a critical aspect of HCI, as it can provide valuable insights into basic human behavior. The gaze-matching method is a reliable approach that can identify the area at which a user is looking. Early methods of gaze tracking required users to wear glasses with a tracking function and limited tracking to a small monitoring area. Additionally, gaze estimation was restricted to a fixed posture within a narrow range. In this study, we proposed a novel non-contact gaze-mapping system that could overcome the physical limitations of previous methods and be applied in real-world environments. Our experimental results demonstrated an average gaze-mapping accuracy of 92.9% across 9 different test environments. Moreover, we introduced the GIST gaze-mapping (GGM) dataset, which served as a valuable resource for learning and evaluating gaze-mapping techniques.

Improved Adaboost-Camshift Face Tracking System in Complex Background

With the rapid growth of science and technology, people pay more and more attention to pattern recognition and computer interaction. Therefore, in the last few years, face detection and tracking technology in video sequence has become a hot topic for people to study. Face tracking detection has a wide application prospect in human-computer interaction, intelligent monitoring, video conference and other aspects. In this paper, the problem of face tracking in video sequences is divided into two aspects: face detection and moving object tracking algorithm. In the face detection problem, the face detection based on Adaboost algorithm is described in detail, and the three-frame difference method is added to make the algorithm better and enhance the speed of face detection. In terms of moving object tracking algorithm, Camshift face tracking algorithm based on color histogram is adopted, which is not affected by the shape and size of the target and has good real-time performance. However, under the influence of color interference and occlusion, the algorithm will make tracking errors. Therefore, Kalman filter is introduced. The algorithm can directly delineate the candidate areas of face to be detected, so as to ensure the feasibility of face tracking. The simulation video image face tracking system is verified by Matlab software. The experimental results show that the system can accurately detect and track the faces in the video image sequence, not only in the simple background, but also in the complex background and multiple faces can also be well detected and tracked, and the tracking ensures real-time performance.

How Engineers’ Imaginaries of Healthcare Shape Design and User Engagement: A Case Study of a Robotics Initiative for Geriatric Healthcare AI Applications

In the development of robotics and Artificial Intelligence (AI) for healthcare, human-centered approaches seek to meet the requirements of healthcare practice and address social and ethical aspects proactively. In this work, an important but neglected aspect of human-computer interaction (HCI) is how engineers understand and envision the healthcare context. Drawing on insights from STS on engineers’ imaginaries and their role in shaping research and development of new technologies, we propose engineers’ imaginaries of healthcare as a point of analysis and intervention for ethical and social aspects of AI and robotics for healthcare. To illustrate the utility of this lens, we use it to on report a case study of an engineering project that develops robotic and AI applications for healthcare. We followed and sought to advance an embedded ethics and social science approach, where ethicists and social scientists accompanied this engineering project using direct interdisciplinary collaboration, observations, and in-depth qualitative interviews with the project’s engineers (n = 18). We analyze how the engineers imagine healthcare as an environment for robots, healthcare workers as potential users, and healthcare practices, and how these imaginaries connect to the design narratives that guide their work. Our findings provide pertinent input for HCI, STS, and engineering ethics related to healthcare AI and robotics, as they speak to prevalent narratives of “assistance” systems, aspects of how human healthcare practices are reframed and valued in the face of new technologies, questions of division of labor between machines and healthcare practitioners, and the implications of ‘acceptance’ as a frame for user-centered design.

Understanding the customer experience in human-computer interaction: a systematic literature review.

In recent years, there has been an increasing interest in customer experience (CX) and its relation to the human-computer interaction (HCI) field. The CX is different depending on the domain in which it is studied, and therefore its dimensions may vary. This research presents an extensive review of 122 studies related to CX definitions and dimensions that have been proposed in different domains, including an analysis from an HCI perspective. The guidelines proposed by Kitchenham & Charters (2007) were used, complementing the review with a snowballing approach. We identified 71 CX definitions (where 14 definitions highlight HCI aspects), 81-dimensional proposals (where 24 proposals contain HCI aspects), and 39 application domains (where 18 domains cover topics related to HCI). However, we did not find CX definitions or dimensions directly focused on HCI. Based on the results, a novel CX definition and dimensions-focused on the HCI area-are proposed and activities that the authors should perform when proposing new CX dimensions in domains related to HCI are suggested, i.e., domains that involve the interaction of a user (or customer) with a software product. Implications for future practice focus on facilitating the understanding of the CX concept and its relationship with HCI; recognizing the key CX dimensions for different domains and how they relate to HCI dimensions; and helping in the creation of new CX dimensions by suggesting activities that can be performed. The results show that there are opportunities for HCI/CX researchers and practitioners to propose new dimensions of CX for a domain related to HCI, develop instruments that allow the evaluation of CX from an HCI point of view, and perform reviews on a particular domain relevant to HCI but less studied.

A Road-Based 3D Navigation System in GIS: A Case Study of an Institute Campus

A user-controlled navigation system is one of the important aspects of human-computer interaction. Finding best path from one location to another and navigating through that path has been a great concern in the geo-virtual navigation. Earlier studies on the geo-virtual navigation systems mainly focus on navigation and visualization, but lack in geo-spatial analysis. Geo-spatial analysis is the domain of geographic information systems (GIS) in which 3D geo-spatial information is used for navigation, geo-visualization, and geo-spatial analysis. The present study deals with wayfinding in the road network of the campus of National Institute of Technology (NIT) Hamirpur, India, in a hilly terrain. It facilitates perform various types of geo-spatial analyses on the road network and virtual travelling in a 3D space.

Fire Hawk Optimizer with Deep Learning Enabled Human Activity Recognition

Human-Computer Interaction (HCI) is a sub-area within computer science focused on the study of the communication between people (users) and computers and the evaluation, implementation, and design of user interfaces for computer systems. HCI has accomplished effective incorporation of the human factors and software engineering of computing systems through the methods and concepts of cognitive science. Usability is an aspect of HCI dedicated to guaranteeing that human–computer communication is, amongst other things, efficient, effective, and sustaining for the user. Simultaneously, Human activity recognition (HAR) aim is to identify actions from a sequence of observations on the activities of subjects and the environmental conditions. The vision-based HAR study is the basis of several applications involving health care, HCI, and video surveillance. This article develops a Fire Hawk Optimizer with Deep Learning Enabled Activity Recognition (FHODL-AR) on HCI driven usability. In the presented FHODL-AR technique, the input images are investigated for the identification of different human activities. For feature extraction, a modified SqueezeNet model is introduced by the inclusion of few bypass connections to the SqueezeNet among Fire modules. Besides, the FHO algorithm is utilized as a hyperparameter optimization algorithm, which in turn boosts the classification performance. To detect and categorize different kinds of activities, probabilistic neural network (PNN) classifier is applied. The experimental validation of the FHODL-AR technique is tested using benchmark datasets, and the outcomes reported the improvements of the FHODL-AR technique over other recent approaches.

Preprocessing Challenges for Real World Affect Recognition

Real world human affect recognition requires immediate attention which is a significant aspect of humancomputer interaction. Audio-visual modalities can make a significant contribution by providing rich contextual information. Preprocessing is an important step in which the relevant information is extracted. It has a crucial impact on prominent feature extraction and further processing. The main aim is to highlight the challenges in preprocessing real world data. The research focuses on experimental testing and comparative analysis for preprocessing using OpenCV, Single Shot MultiBox Detector (SSD), DLib, Multi-Task Cascaded Convolutional Neural Networks (MTCNN), and RetinaFace detectors. The comparative analysis shows that MTCNN and RetinaFace give better performance in real world data. The performance of facial affect recognition using a pre-trained CNN model is analysed with a lab-controlled dataset CK+ and a representative wild dataset AFEW. This comparative analysis demonstrates the impact of preprocessing issues on feature engineering framework in real world affect recognition.

Data Analytics and Visualization Application for Asset Health Monitoring

Much of the research on predictive maintenance has focused on statistical and machine learning techniques, while there has been significantly less focus on the human computer interaction or visualization aspects of PHM. Human computer interaction and visualization techniques can quickly help identify interesting data sub-domains, both time periods and sensors, provided the data can be queried, retrieved and displayed in a timely manner. Augmenting visualization and interaction with a visual, aggregative fleet-based query system adds a further dimension, highlighting the ability of the fleet to carry out its mission. In the age of big data, effectively visualizing data from an asset with a multitude of sensors can be difficult, and that difficulty is compounded as you add additional assets.
 In this paper, we propose a scalable framework that is capable of visualizing past, current, and prognosticated health from the individual sensor up to the fleet or group level. In addition to viewing near real time sensor data, maintenance logs, fault information, and data aggregations will be merged with the sensor data to make the analysis and visualizations more valuable. This framework is scalable in regard to how much data can be collected, stored, and processed, as well as to the different organizational levels within a fleet of assets. The framework is built as a web-application primarily using the following visualizations: a collapsible tree graph for asset information; 2D charts for temporal sensor data, fault data, and maintenance data; and 3D digital twins of critical components. 
 The current dataset used to build and demonstrate the capability of the web-application has data from over 3000 vehicles and contains approximately 9TB of data. Vehicle information such as model, make, sub-component, and fleet organization are presented in a configurable, collapsible tree structure. This allows the user to visualize the fleet and to select the asset and sensor combinations needed to display the temporal sensor data that will answer the question at hand. Information regarding each vehicle’s health status is displayed then aggregated and displayed for each of the higher tree nodes. A 3D digital twin is also available to highlight sensor locations and current health status of the component. These component models can be viewed and manipulated with and without a virtual reality headset. As health status monitoring for asset subcomponents are developed, they can be added to the system, allowing for complete health status reporting.

BehavePassDB: Public Database for Mobile Behavioral Biometrics and Benchmark Evaluation

Mobile behavioral biometrics have become a popular topic of research, reaching promising results in terms of authentication, exploiting a multimodal combination of touchscreen and background sensor data. However, there is no way of knowing whether state-of-the-art classifiers in the literature can distinguish between the notion of user and device. In this article, we present a new database, BehavePassDB, structured into separate acquisition sessions and tasks to mimic the most common aspects of mobile Human-Computer Interaction (HCI). BehavePassDB is acquired through a dedicated mobile app installed on the subjects devices, also including the case of different users on the same device for evaluation. We propose a standard experimental protocol and benchmark for the research community to perform a fair comparison of novel approaches with the state of the art11https://github.com/BiDAlab/MobileB2C_BehavePassDB/.. We propose and evaluate a system based on Long-Short Term Memory (LSTM) architecture with triplet loss and modality fusion at score level.