Area Of Human-computer Interaction Research Articles

Multifaceted formal methods and their interdisciplinary role — From the cathedral of ‘components as coalgebras’ to the HCI context and the open source software bazaar

In this article we revisit the history of formal methods with a focus on important aspects that contribute to their interdisciplinary role. We consider: the variability of mathematical representation techniques on which the theoretical foundations of formal methods are based; formal methods multidisciplinarity; their capability to serve at a meta-level in providing the semantics of programming languages, specification and modelling languages as well as higher-level and domain-specific formal notations; and, finally, how some of these higher-level and domain-specific notations may be lifted at an interdisciplinary level.Within this historical review, we are inspired by Luís Barbosa's “components as coalgebras” approach in seeing that the duality data-process is underlying all those aspects of formal methods. We also see that such a duality may not only be expressed in universal terms within category theory, but may also be characterised in practical terms and focused applications by two distinct logic paradigms, equational logic for the data and rewriting logic for the process, by two modelling directions, forward process definitions and backward data-driven process transformations, and by the distinction between syntax, defined by the data structures, and semantics, provided by rewrite rules. We use the Maude modelling language to illustrate the application of the data-process duality. In fact, Maude use equational logic to define data types and rewriting logic to express system evolution. Illustrative examples are from the areas of cognitive science and human-computer interaction (HCI).We then define a data-driven model transformation, which we call elaborative mining, which adopts a backward perspective to recover a behaviour that was observed in real life but was not predicted forward by the original model. Finally, we see how the “open source software bazaar”, which is a metaphor for the apparently chaotic open source development process, offers us a big data context to lift the driving process for model transformation from deterministic to statistical.

Fabrication of flexible pressure sensor for human detection by direct-write printing concave surface

Flexible pressure sensor plays an important role in the field of human-computer interaction area due to its high flexibility and sensitivity. Convex surface of flexible electrodes provide an efficient way to enhance the sensitivity of flexible pressure sensors. However, facilely preparing controllable convex surface of flexible electrodes is still a challenge for obtaining high sensitive flexible pressure sensors. In this paper, direct-write print technique was utilized to prepare a concave structure surface with viscoelastic substrate, which could be used to prepare convex structure flexible electrode for fabricating highly sensitive flexible pressure sensors. Interaction between ink droplet and substrate was explored to generate a concave template, which could be controlled in spacing and diameter with printing spacing and nozzle diameter. Convex structure flexible surface could be prepared with the concave template, and then the convex structure flexible electrode could be realized with the evaporation of metallic silver on the surface. Furthermore, flexible pressure sensor was assembled with an interlocking structure by stacking two convex structure flexible electrodes in a face-to-face way, which could efficiently enhance flexibility and electrical property. The flexible pressure sensor presents a high sensitivity, a fast response/recovery time (<100 ms) and excellent flexibility (more than 10,000 cycles of bending), which could be applied in monitor physiological signals such as pulse detection, sound vibration, finger bending and so on. Therefore, this work provides an efficient method for preparing controllable structured surface, which has a great potential in the fabrication of flexible sensors, wearable electronics and energy devices.

Exploring the impact of HCI on smart buildings: A comprehensive literature review on user experience and sustainability

This paper provides insights into the application of human-computer interaction (HCI) principles in the field of smart building design, particularly their impact on improving occupant comfort, building energy efficiency, and the user interaction experience. The research context focuses on advances in HCI and the importance of smart buildings in enhancing quality of life and sustainability. The research theme focuses on how HCI principles can be utilized to improve the user interface design of smart buildings in order to promote more effective user-building system interactions. Through an extensive review of existing literature in the area of HCI and smart buildings, including recent research papers, case study reports, and technical overviews, this study concentrates on analyzing examples of HCI applications in smart lighting, temperature control systems, and energy management. Through this comprehensive literature review, this paper aims to illustrate how HCI principles can provide theoretical and practical guidance for the design of more humanized, interactive and environmentally friendly smart buildings, thereby providing insights and recommendations for future smart building design and research.

The future of interactive information radiators for knowledge workers

Abstract Information Radiators (IRs) provide context-specific pieces of information in a semi-public place where a group of people can see it while working or passing-by. They can simplify information sharing “out-of-the-box”, foster awareness and socialization, create serendipity and enhance collaboration. Recent sociotechnical developments such as the establishment of permanent hybrid work settings as well as advances in the area of Human Computer Interaction (HCI) such as the emergence of Augmented Reality (AR) and Virtual Reality (VR) are likely to impact how IRs are being used – or even challenge their usefulness. In this article we discuss those developments and their possible implications for the design and use of IRs in the context of knowledge work in the next decades. We argue that IRs will probably remain an important part of future office environments providing awareness, supporting serendipity and building a situated social place for matchmaking as well as informal communication. Using new display and interaction technologies (such as AR) they might even grow in importance by enabling fluid work scenarios.

On the Use of Pitch-Based Features for Detecting Simultaneous Fear Emotion and Deception Behavior From Speech

In the last years, endowing the machine with emotional and behavioral intelligence has been one of the most challenging issues in the human–computer interaction area. In this context, this paper investigates the detection of simultaneous fear emotion and deception behavior in speech. To do so, a set of 72 pitch-based features has been investigated first to recognize fear and deception separately. Then, different feature selection techniques have been used in order to select the most relevant ones that best discriminate between fear/nonfear and deception/nondeception classes. Next, a decision-level fusion approach based on the belief theory has been proposed to infer whether fear and deception are detected simultaneously. Simulation results carried on databases dealing with fear/nonfear emotions and deception/truth behaviors have shown classification results reaching 83.33% and 72.45% as accuracy rates for fear and deception classifiers, respectively. The proposed fusion approach has revealed a correspondence between fear emotion and deception behavior in speech modality.

MoS2-based charge trapping layer enabled triboelectric nanogenerator with assistance of CNN-GRU model for intelligent perception

Self-powered sensing technology and smart perception technology have broad application prospects in flexible and wearable electronics. In this work, a flexible triboelectric nanogenerator (RMP-TENG) based on a room-temperature vulcanized silicone rubber (RTV)@(Molybdenum disulfide (MoS2)/Polyvinyl chloride (PVC)) functional layer is developed using a layer-by-layer self-assembly and material doping strategy. The RTV@(MoS2/PVC) functional layer is divided into a charge-generating layer (RTV/PVC) and a charge-trapping layer (RTV/MoS2). The synergistic effect of PVC and MoS2 has improved the surface roughness and charge transfer efficiency of RMP-TENG, doubling its output performance to 1055 V and 112 μA. In order to further improve the tactile sensing accuracy of RMP-TENG, a deep learning model based on Convolutional Neural Network (CNN) and Gated Recurrent Unit (GRU) is developed. It predicts the type of contact material based on the features of tactile signals, achieving a prediction accuracy of 93.975%. Additionally, by combining mobile smart terminals, the CNN-GRU model, and RMP-TENG, a wireless access control system based on self-powered tactile sensing and intelligent material recognition is developed. Through the optimization of these experiments and algorithms, RMP-TENG has achieved real-time material recognition capabilities. This demonstrates the broad application prospects of RMP-TENG in wearable energy supply, intelligent sensing, human-computer interaction, and other areas.

An Analysis of the Authorship and Co-authorship Networks of the Brazilian Human-Computer Interaction Conference

In Brazil, the Brazilian Symposium on Human Factors in Computing Systems (IHC) gathers the scientific community of researchers interested in the field of Human-Computer Interaction since 1998, being the main Brazilian event in this sub-area of Computing. Over twenty-one editions, the IHC received works from researchers from different regions of the country who, over the years, have been building their own co-authorship relationships with the other authors of the Symposium. In this context, this paper analysed the IHC from the perspective of those who helped to consolidate this important national scientific event, as well as in the expansion of the Human-Computer Interaction area in the Brazilian scenario, that is, its researchers-authors. In total, 1,443 authors were identified and analysed in the study presented in this work, which considered 873 publications of three IHC tracks: Full Papers, Short Papers, and Innovative Ideas and Emerging Results. Issues related to the publications and to the co-authorship relationships of these authors over the years and in the different article tracks of the IHC were considered. In order to describe their research trajectories within the IHC itself, the study presents, in different scales of time, how these authors evolved in relation to their contributions over time. In addition, this paper analyses how the authors contributed with each other and originated the complex collaboration network of the IHC. For this, co-authorship networks and groups of authors who published together were explored, aiming to clarify the collaborations between these authors, as well as how they evolved until the edition of 2022. In this sense, this work seeks, with each research question, to simplify the presentation of results through different visualizations, which were planned and created to describe information that are not clearly evident when observing the IHC publications in a “disconnected” manner. The results of this study are revealed, described and analysed under different perspectives, as well as discussed in details in this paper.

CLASSIFICATION OF COGNITIVE STRESS AS A PSYCHOLOGICAL INDICATOR THROUGH MACHINE LEARNING

In this study, we explored the potential of Support Vector Machine (SVM) method for classifying levels of cognitive stress using EEG (Electroencephalogram) signals. The goal is to develop accurate models that would enable the prediction and understanding of not only the current mental state of the subjects, but also potential real-time interventions. In medical fields, the application can be seen in the treatment of attention, focus, hyperactivity, autism, and depression disorders. Additionally, there is an extremely high potential for application in areas such as psychology, sociology, education, economics, neuromarketing, security, and in enhancing workplace stress management, anxiety treatment, digital marketing, economicfinancial forensics, as well as improving user experience in virtual environments and video games. The results have shown that it is possible to differentiate high and low levels of cognitive stress with satisfactory accuracy, opening the way for the application of these findings in various fields. Cognitive stress represents one of the fundamental cognitive processes that causes individuals to behave and think differently in certain situations than in their usual state of consciousness. Predicting, analyzing, and understanding the level of cognitive stress from EEG signals is of great importance in various fields, including neuroscience, psychology, education, professional sports, human-computer interaction, and many other areas. Machine learning represents a subgroup of artificial intelligence that uses statistical models, and functions to ‘learn’ and ‘train’ data resulting in corresponding output values. The brain-computer interface, through which data on cognitive stress, among other parameters and psychological categories, is collected, is based on the functioning of EEG devices. The prediction of cognitive stress represents the application of machine learning, recording and using brain EEG signals or extracted characteristics from EEG signals as input values, in order to predict the level of output values of cognitive stress, of high or low degree, reflecting the mental state of the subjects in real time

I Know This Looks Bad, But I Can Explain: Understanding When AI Should Explain Actions In Human-AI Teams

Explanation of artificial intelligence (AI) decision-making has become an important research area in human–computer interaction (HCI) and computer-supported teamwork research. While plenty of research has investigated AI explanations with an intent to improve AI transparency and human trust in AI, how AI explanations function in teaming environments remains unclear. Given that a major benefit of AI giving explanations is to increase human trust understanding how AI explanations impact human trust is crucial to effective human-AI teamwork. An online experiment was conducted with 156 participants to explore this question by examining how a teammate’s explanations impact the perceived trust of the teammate and the effectiveness of the team and how these impacts vary based on whether the teammate is a human or an AI. This study shows that explanations facilitate trust in AI teammates when explaining why AI disobeyed humans’ orders but hindered trust when explaining why an AI lied to humans. In addition, participants’ personal characteristics (e.g., their gender and the individual’s ethical framework) impacted their perceptions of AI teammates both directly and indirectly in different scenarios. Our study contributes to interactive intelligent systems and HCI by shedding light on how an AI teammate’s actions and corresponding explanations are perceived by humans while identifying factors that impact trust and perceived effectiveness. This work provides an initial understanding of AI explanations in human-AI teams, which can be used for future research to build upon in exploring AI explanation implementation in collaborative environments.

A review of motion generation technology

Nowadays, deep learning and neural network-related research play a very important role in the widely use of artificial intelligence -related technologies, Among them, the hot development in the direction of generative adversarial networks (GAN) has given birth to many generation-related techniques. For example, MoCoGAN is based on the implementation principle of GAN, which enables video generation of different actions of the same character or the same action of different characters, through the method that decompose video into actions and content. This paper introduces the history and principles of MoCoGAN, starting from the prospect of using MoCoGAN in artificial intelligence (AI) industry and the technical challenges that need to be overcome in the future application of action generation. Besides, this paper also discusses the two main issues of how to improve the quality of video generation using MoCoGAN and the input conditions that are the most central problem to GAN networks. By summarizing the optimization solutions of other researchers in these two areas in recent years, this paper searches the core problems need to be solved and propose a broad prospect for future video generation techniques that can be implemented by using MoCoGAN in human-computer interaction (HCI) area.

Posture and stress detection system using open CV and media pipe

Posture and stress are two critical factors affecting a person's physical and mental well-being. Proper posture and stress management can help avoid a range of health issues. Poor posture may result in chronic discomfort, decreased mobility, and an increased risk of musculoskeletal problems. Similarly, stress can negatively impact physical and mental health, contributing to conditions such as depression, anxiety, and cardiovascular disease. Traditional methods for assessing posture and stress, including physical examinations or self-reporting, can be subjective and time-consuming. Recent advances in machine learning and computer vision techniques have enabled the development of models that automatically detect posture and stress levels from video data. In this study, a Django framework was built that incorporates models to assess posture by calculating angles between tracked distance vectors. Stress levels are evaluated based on facial features and expressions. The use of skeleton data for human posture recognition is a key research area in human-computer interaction. By employing the MSR 3D Action Dataset, 33 key skeletal points on the body are detected, aiding in posture determination. Additionally, analyzing facial features and emotions is essential for estimating stress levels. The approach relies on convolutional neural networks.

Deep learning-based facial emotion recognition exploiting self-supervised pre-training and feature pyramid

Facial emotion recognition (FER), a technology designed to automatically identify the emotional state of an individual based on the features of his face, is a thriving area in human-computer interaction and affective computing. Among all the FER techniques, the deep learning model, especially the convolutional neural network is more successful. In this paper, various networks are tested firstly with different optimizers based on the VGG baseline. After that, according to the results of the test, it seems that VGG19 is the best choice for the backbone network. Following that, a feature pyramid network added to the backbone model is considered to improve the model, which, however, ends up with even worse accuracy. To find another way of improvement, the attention is thrown to the unsupervised contrastive model, which is built based on the momentum contrast with VGG19 being the backbone network, culminating in an increase in the accuracy from 71.94% to 72.11% in the end.

The Development and Opportunity of Educational Mode in the Metaverse Age

The Metaverse is an end-users-oriented integration of various layers of Information Technology (IT), where Human–Computer Interaction (HCI) would be the core technology. With the rapid development of IT, the Metaverse would allow users to connect, work, conduct business, and access educational resources, all in a technology-mediated environment in new interaction ways. The Metaverse can play a major role in the future of online learning and enable a rich active learning environment, where learners have the opportunity to obtain first-hand experiences that might not be accessible in the physical world. While currently there is a severe shortage in Metaverse-Learning studies, such research strands are expected to soon emerge. The main objective of this paper is to investigate challenges and opportunities for human-centric Metaverse technology in the learning sector, hence accelerating research in this field. A phenomenological research method was used, including semi-structured in-depth interviews, essays written by participants, a focus group discussion with 19 experts in the areas of HCI, intelligent interactive technologies, and online learning. Five challenges were identified for the Metaverse-Learning context: immersive design, privacy and security, universal access, physical and psychological health concerns, and governance. While the research provided suggestions to overcome those challenges, three Meta-Learning opportunities were identified: hands-on training and learning, game-based learning, and collaboration in creating knowledge. The findings of this research contribute to understanding the complexity of the online learning in the Metaverse from the Human–Computer Interaction point of view. These findings can be used to further research the Metaverse as a virtual communication environment and potential business and learning platform.

AI in Human-Computer Interaction

The emergence of human-computer interaction as a concept can be attributed to the progress made in computer technology. Educated and technically proficient young adults often take part in research experiments related to human-computer interaction. The mental model in human-computer interaction is the main topic of this essay. One of the tactics used in this review study is to call attention to current theories, findings, and advancements in HCI. A second strategy is to look for study areas that haven't been explored yet but are falling behind. In order to produce better user-friendly products, this study emphasizes the importance of fidelity prototyping, which highlights the emotional intelligence of the user. This process is still being developed and designed as an automated method. The area of human-computer interaction explores the interactions between people and computers, focusing on the extent to which computers are designed to effectively accommodate human needs and preferences. A user- friendly method of computer access is speech and pattern recognition. The suggested system will launch the speech driver before capturing user input using the microphone. We'll translate the speech input into text. The text input is transformed into tokens by the lexical analyzer and saved in the symbol table. Tokens are fed into the parser, which creates a parse tree. This vocal input can be used to execute various Linux OS commands. The speech will be analyzed for keywords that apply to other activities, and the corresponding command will be carried out. The suggested system will function in both terminal and word processor modes. It will carry out both internal and external commands in terminal mode. When in word processor mode, it will carry out actions like open, type, save, and exit. Operations will be carried out by the system in response to oral commands. Keywords— Human Computer Interaction, Natural Language Processing, Application Program Interface, Speech to text, Emotional intelligence, Interactivity, Younger participants, Fidelity Prototyping.

Advances in electronic skin research: a bibliometric analysis

Background: E-skin (electronic skin) is an active research area in human-computer interaction and artificial intelligence.Methods: A bibliometric analysis was performed to evaluate publications in the E-skin field between 2000 and 2021 based on the Web of Science (WoS) databases.Results: A total of 4,954 documents were identified. A detailed overview of E-skin research was presented from aspects of productive countries/regions, institutions, journals, citations, highly cited papers, keywords, and emerging topics. With the emergence of new functional materials, structural design, 3D printing, and nanofabrication techniques, E-skin research has achieved dramatic progress after 2013. Scholars and institutions in China, the United States and South Korea are leading the way in E-skin research. Pressure sensor, strain sensor, and flexible electronics are the most focused directions at present and Internet of things is the most emerging topic.Conclusion: E-skin research has achieved dramatic progress but there is still quite a challenging task in practical applications. Manufacturing process simplification, cost reduction, functional integration, energy supply, and biocompatibility are vital for the future development of E-skin.

Toward Facial Expression Recognition in the Wild via Noise-Tolerant Network

Facial Expression Recognition (FER) has recently emerged as a crucial area in Human-Computer Interaction (HCI) system for understanding the user’s inner state and intention. However, feature- and label-noise constitute the major challenge for FER in the wild due to the ambiguity of facial expressions worsened by low-quality images. To deal with this problem, in this paper, we propose a simple but effective <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Facial Expression Noise-tolerant Network (FENN)</i> which explores the inter-class correlations for mitigating ambiguity that usually happens between morphologically similar classes. Specifically, FENN leverages a multivariate normal distribution to model such correlations at the final hidden layer of the neural network to suppress the heteroscedastic uncertainty caused by inter-class label noise. Furthermore, the discriminative ability of deep features is weakened by the subtle differences between expressions and the presence of feature noise. FENN utilizes a feature-noise mitigation module to extract compact intra-class feature representations under feature noise while preserving the intrinsic inter-class relationships. We conduct extensive experiments to evaluate the effectiveness of FENN on both original annotated images and synthetic noisy annotated images from RAF-DB, AffectNet, and FERPlus in-the-wild facial expression datasets. The results show that FENN significantly outperforms state-of-the-art FER methods.

Multi-branch feature learning based speech emotion recognition using SCAR-NET

Speech emotion recognition (SER) is an active research area in affective computing. Recognizing emotions from speech signals helps to assess human behaviour, which has promising applications in the area of human-computer interaction. The performance of deep learning-based SER methods relies heavily on feature learning. In this paper, we propose SCAR-NET, an improved convolutional neural network, to extract emotional features from speech signals and implement classification. This work includes two main parts: First, we extract spectral, temporal, and spectral-temporal correlation features through three parallel paths; and then split-convolve-aggregate residual blocks are designed for multi-branch deep feature learning. The features are refined by global average pooling (GAP) and pass through a softmax classifier to generate predictions for different emotions. We also conduct a series of experiments to evaluate the robustness and effectiveness of SCAR-NET which can achieve 96.45%, 83.13%, and 89.93% accuracy on the speech emotion datasets EMO-DB, SAVEE, and RAVDESS. These results show the outperformance of SCAR-NET.

Metaverse-Based Learning Opportunities and Challenges: A Phenomenological Metaverse Human–Computer Interaction Study

The Metaverse is an end-users-oriented integration of various layers of Information Technology (IT), where Human–Computer Interaction (HCI) would be the core technology. With the rapid development of IT, the Metaverse would allow users to connect, work, conduct business, and access educational resources, all in a technology-mediated environment in new interaction ways. The Metaverse can play a major role in the future of online learning and enable a rich active learning environment, where learners have the opportunity to obtain first-hand experiences that might not be accessible in the physical world. While currently there is a severe shortage in Metaverse-Learning studies, such research strands are expected to soon emerge. The main objective of this paper is to investigate challenges and opportunities for human-centric Metaverse technology in the learning sector, hence accelerating research in this field. A phenomenological research method was used, including semi-structured in-depth interviews, essays written by participants, a focus group discussion with 19 experts in the areas of HCI, intelligent interactive technologies, and online learning. The individual interviews took place in May 2022, with a focus group meeting held online in June 2022 to formulate a collective opinion of the 19 experts. Five challenges were identified for the Metaverse-Learning context: immersive design, privacy and security, universal access, physical and psychological health concerns, and governance. While the research provided suggestions to overcome those challenges, three Meta-Learning opportunities were identified: hands-on training and learning, game-based learning, and collaboration in creating knowledge. The findings of this research contribute to understanding the complexity of the online learning in the Metaverse from the Human–Computer Interaction point of view. These findings can be used to further research the Metaverse as a virtual communication environment and potential business and learning platform.

Human-Computer Integration as an Extension of Interaction: Understanding Its State-of-the-Art and the Next Challenges

The Interaction Era is evolving and extending to the Human-Computer Integration Era, in which the relationship between humans and technologies is characterized as a partnership. Motivated by this evolution, we conducted a Systematic Literature Review to present the current state of knowledge about the emerging paradigm of Human-Computer Integration (HInt), as well as its opportunities and challenges for the Human-Computer Interaction (HCI) area. As a result, we organize the existing knowledge and research on HInt which describes it as a new interaction paradigm. Our findings indicate that HInt is a topic of growing interest in the HCI Community, and provide an overview of existing initiatives and key research gaps on HInt within the HCI scope. As contributions, our results are useful to (1) expand the HCI community’s knowledge regarding this emerging paradigm and (2) stimulate new initiatives and contributions to the consolidation and evolution of the HCI with a lens on HInt.

Semantic-Rich Facial Emotional Expression Recognition

The ability to perceive human facial emotions is an essential feature of various multi-modal applications, especially in the intelligent human-computer interaction (HCI) area. In recent decades, considerable efforts have been put into researching automatic facial emotion recognition (FER). However, most of the existing FER methods only focus on either basic emotions such as the seven/eight categories (e.g., <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">happiness, anger</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">surprise</i> ) or abstract dimensions ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">valence, arousal, etc.</i> ), while neglecting the fruitful nature of emotion statements. In real-world scenarios, there is definitely a larger vocabulary for describing human's inner feelings as well as their reflection on facial expressions. In this work, we propose to address the semantic richness issue in the FER problem, with an emphasis on the granularity of the emotion concepts. Particularly, we take inspiration from former psycho-linguistic research, which conducted a prototypicality rating study and chose 135 emotion names from hundreds of English emotion terms. Based on the 135 emotion categories, we investigate the corresponding facial expressions by collecting a large-scale 135-class FER image dataset and propose a consequent facial emotion recognition framework. To demonstrate the accessibility of prompting FER research to a fine-grained level, we conduct extensive evaluations on the dataset credibility and the accompanying baseline classification model. The qualitative and quantitative results prove that the problem is meaningful and our solution is effective. To the best of our knowledge, this is the first work aimed at exploiting such a large semantic space for emotion representation in the FER problem.