Human-computer Interaction Studies Research Articles

From explainable to interactive AI: A literature review on current trends in human-AI interaction

AI systems are increasingly being adopted across various domains and application areas. With this surge, there is a growing research focus and societal concern for actively involving humans in developing, operating, and adopting these systems. Despite this concern, most existing literature on AI and Human–Computer Interaction (HCI) primarily focuses on explaining how AI systems operate and, at times, allowing users to contest AI decisions. Existing studies often overlook more impactful forms of user interaction with AI systems, such as giving users agency beyond contestability and enabling them to adapt and even co-design the AI’s internal mechanics. In this survey, we aim to bridge this gap by reviewing the state-of-the-art in Human-Centered AI literature, the domain where AI and HCI studies converge, extending past Explainable and Contestable AI, delving into the Interactive AI and beyond. Our analysis contributes to shaping the trajectory of future Interactive AI design and advocates for a more user-centric approach that provides users with greater agency, fostering not only their understanding of AI’s workings but also their active engagement in its development and evolution.

Who Made That Decision and Why? Users’ Perceptions of Human Versus AI Decision-Making and the Power of Explainable-AI

With the advent of artificial intelligence (AI) based systems, a new era has begun. Decisions that were once made by humans are now increasingly being made by these advanced systems, with the inevitable consequence of our growing reliance on AI in many aspects of our lives. At the same time, the opaque nature of AI-based systems and the possibility of unintentional or hidden discriminatory practices and biases raises profound questions not only about the mechanics of AI, but also about how users perceive the fairness of these systems. We hypothesize that providing various explanations for AI decision-making processes and output may enhance users’ fairness perceptions and make them trust the system and adopt its decisions. Hence, we devised an online between-subject experiment that explores users’ fairness and comprehension perceptions of AI systems with respect to the explanations provided by the system, employing a case study of a managerial decision in the human resources (HR) domain. We manipulated (i) the decision-maker (AI or human); (ii) the input (candidate characteristics); (iii) the output (recommendation valence), and (iv) the explanation style. We examined the effect of the various manipulations (and individuals’ demographic and personality characteristics) using multivariate ordinal regression. We also performed a multi-level analysis of experiment components to examine the effects of the decision-maker type, explanation style, and their combination. The results suggest three main conclusions. The first conclusion is that there is a gap in users’ fairness and comprehension perception of AI-based decision making systems compared to human decision making. The second conclusion is that knowing that an AI-based system provided the decisions negatively affects users’ fairness and comprehension perceptions, compared to knowing that humans made the decision. Finally, the third conclusion is that providing case-based, certification-based, or sensitivity-based explanations can narrow this gap and may even eliminate it. Additionally, we found that users’ fairness and comprehension perceptions are influenced by a variety of factors such as the input, output, and explanation provided by the system, as well as by individuals’ age, education, computer skills, and personality. Our findings may help to understand when and how to use explanations to improve users’ perceptions regarding AI-based decision-making. CCS CONCEPTS • Human computer interaction (HCI) → HCI design and evaluation methods → User studies • Human-centered computing → Human computer interaction (HCI) → Empirical studies in HCI • Applied computing → Law, social and behavioral sciences → Sociology

Electroencephalographic Signal Data Augmentation Based on Improved Generative Adversarial Network.

EEG signals combined with deep learning play an important role in the study of human-computer interaction. However, the limited dataset makes it challenging to study EEG signals using deep learning methods. Inspired by the GAN network in image generation, this paper presents an improved generative adversarial network model L-C-WGAN-GP to generate artificial EEG data to augment training sets and improve the application of BCI in various fields. The generator consists of a long short-term memory (LSTM) network and the discriminator consists of a convolutional neural network (CNN) which uses the gradient penalty-based Wasserstein distance as the loss function in model training. The model can learn the statistical features of EEG signals and generate EEG data that approximate real samples. In addition, the performance of the compressed sensing reconstruction model can be improved by using augmented datasets. Experiments show that, compared with the existing advanced data amplification techniques, the proposed model produces EEG signals closer to the real EEG signals as measured by RMSE, FD and WTD indicators. In addition, in the compressed reconstruction of EEG signals, adding the new data reduces the loss by about 15% compared with the original data, which greatly improves the reconstruction accuracy of the EEG signals' compressed sensing.

Interaction mode enables user perception recognition and perception optimization: An AI human-computer interaction study

Artificial intelligent (AI) has various ways of human-computer interaction, but most of them overlook the recognition of human perception. If the interaction mode is combined with psychology, the user's mood change can be identified by the user's subtle expression, movement change and voice tone change, so as to provide corresponding services and improve the user experience. Statistical analysis of human responses to different situations in cognitive psychology, incorporating them into human-computer interaction methods. The current human-computer interaction modes in products tend to be standardized, and focusing user experience on user perception will bring special experiences to users. Emotional recognition is a cross disciplinary discipline with broad application prospects, but it has not yet reached a mature stage and requires corpus enrichment, theoretical strengthening, and method innovation. The era of artificial intelligence is leading a new wave of technological progress, and emotion recognition, as an important topic in the field of artificial intelligence, can help computer intelligence recognize human emotions and make human-computer interaction more friendly. In the near future, research on emotion recognition technology will make greater progress and be better applied to practical products.

Emotion recognition from speech signals using digital features optimization by diversity measure fusion

Emotion recognition from speech signals serves a crucial role in human-computer interaction and behavioral studies. The task, however, presents significant challenges due to the high dimensionality and noisy nature of speech data. This article presents a comprehensive study and analysis of a novel approach, “Digital Features Optimization by Diversity Measure Fusion (DFOFDM)”, aimed at addressing these challenges. The paper begins by elucidating the necessity for improved emotion recognition methods, followed by a detailed introduction to DFOFDM. This approach employs acoustic and spectral features from speech signals, coupled with an optimized feature selection process using a fusion of diversity measures. The study’s central method involves a Cuckoo Search-based classification strategy, which is tailored for this multi-label problem. The performance of the proposed DFOFDM approach is evaluated extensively. Emotion labels such as ‘Angry’, ‘Happy’, and ‘Neutral’ showed a precision rate over 92%, while other emotions fell within the range of 87% to 90%. Similar performance was observed in terms of recall, with most emotions falling within the 90% to 95% range. The F-Score, another crucial metric, also reflected comparable statistics for each label. Notably, the DFOFDM model showed resilience to label imbalances and noise in speech data, crucial for real-world applications. When compared with a contemporary model, “Transfer Subspace Learning by Least Square Loss (TSLSL)”, DFOFDM displayed superior results across various evaluation metrics, indicating a promising improvement in the field of speech emotion recognition. In terms of computational complexity, DFOFDM demonstrated effective scalability, providing a feasible solution for large-scale applications. Despite its effectiveness, the study acknowledges the potential limitations of the DFOFDM, which might influence its performance on certain types of real-world data. The findings underline the potential of DFOFDM in advancing emotion recognition techniques, indicating the necessity for further research.

Skeleton action recognition via graph convolutional network with self-attention module

<abstract><p>Skeleton-based action recognition is an important but challenging task in the study of video understanding and human-computer interaction. However, existing methods suffer from two deficiencies. On the one hand, most methods usually involve manually designed convolution kernel which cannot capture spatial-temporal joint dependencies of complex regions. On the other hand, some methods just use the self-attention mechanism, ignoring its theoretical explanation. In this paper, we proposed a unified spatio-temporal graph convolutional network with a self-attention mechanism (SA-GCN) for low-quality motion video data with fixed viewing angle. SA-GCN can extract features efficiently by learning weights between joint points of different scales. Specifically, the proposed self-attention mechanism is end-to-end with mapping strategy for different nodes, which not only characterizes the multi-scale dependencies of joints, but also integrates the structural features of the graph and an ability of self-learning fusion features. Moreover, the attention mechanism proposed in this paper can be theoretically explained by GCN to some extent, which is usually not considered in most existing models. Extensive experiments on two widely used datasets, NTU-60 RGB+D and NTU-120 RGB+D, demonstrated that SA-GCN significantly outperforms a series of existing mainstream approaches in terms of accuracy.</p></abstract>

SC2EGSet: StarCraft II Esport Replay and Game-state Dataset

As a relatively new form of sport, esports offers unparalleled data availability. Our work aims to open esports to a broader scientific community by supplying raw and pre-processed files from StarCraft II esports tournaments. These files can be used in statistical and machine learning modeling tasks and compared to laboratory-based measurements. Additionally, we open-sourced and published all the custom tools that were developed in the process of creating our dataset. These tools include PyTorch and PyTorch Lightning API abstractions to load and model the data. Our dataset contains replays from major and premiere StarCraft II tournaments since 2016. We processed 55 “replaypacks” that contained 17930 files with game-state information. Our dataset is one of the few large publicly available sources of StarCraft II data upon its publication. Analysis of the extracted data holds promise for further Artificial Intelligence (AI), Machine Learning (ML), psychological, Human-Computer Interaction (HCI), and sports-related studies in a variety of supervised and self-supervised tasks.

The more-than-human turn in human-plant interaction design: From utilitarian object to living co-inhabitant

Plants and specifically indoor pot plants have featured as a prominent object in human-computer interaction (HCI) studies for more than two decades. Motivated by recent discourse in posthumanist and entanglement HCI, this paper seeks to contribute to the more-than-human turn in interaction design. It uses a twofold approach. We first engage in theoretically grounded inductive reasoning to postulate that the role and agency of plants in interaction design projects ought to change. This is backed up with a deductive approach based on a critical meta-analysis using a methodical review of human-plant interaction studies. The analysis identified a range of characteristics that we discuss in relation to study motives and outcomes. Our findings suggest that so far, plants-as-objects in interaction design are used as (i) a proxy for nature; (ii) a trigger for human experiences or; (iii) an interface for other actions. Our synthesis combining theory and meta-analysis contributes to ongoing design discourse on the ethical and societal implications of using plants in technologically mediated environments. We offer reflective remarks on how HCI can re-conceptualise plants from utilitarian objects towards recognising their value as living co-inhabitants.

Экспериментальное сравнение доверия искусственному интеллекту и человеку в экономических решениях

<p>AI as a new direction in the study of human-computer interaction requires a new look at trust as a phenomenon. In our study, we focus on examining trust in the context of economic behavior. The study took place in two stages. At the first stage, during the interview, we have identified the main factors of trust and mistrust in AI and the specific factors of trust in AI in economic decisions. Also, we have revealed a subjective indicator of the level of trust in the advisor’s recommendations - the economic activity of the participant when performing the recommended action. At the second stage, an experiment was carried out. The participants were asked to play a stock exchange game. The goal of the game was to make money by buying and selling shares. There were an option to ask an advise. For the experimental group, AI acted as an advisor, for the control group, a person (an expert in trading). According to the analysis of 800 economic decisions, economic activity during the game was higher among the participants in the control group who followed the advice of the person (t = 3.646, p <0.001). As a result of the study, three main conclusions were obtained: 1) the level of trust in councils in an economic decision can be expressed in the form of economic activity; 2) the level of trust in economic recommendation depends on whether the recommendation is made by a human or an AI; 3) the specific factors of trust in economic decisions are highlighted: the individuality of the council and the speed of the requested solution.</p>

Context Mining with Machine Learning Approach: Understanding, Sensing, Categorizing, and Analyzing Context Parameters

Context is a vital concept in various fields, such as linguistics, psychology, and computer science. It refers to the background, environment, or situation in which an event, action, or idea occurs or exists. Categorization of context involves grouping contexts into different types or classes based on shared characteristics. Physical context, social context, cultural context, temporal context, and cognitive context are a few categories under which context can be divided. Each type of context plays a significant role in shaping our understanding and interpretation of events or actions. Understanding and categorizing context is essential for many applications, such as natural language processing, human-computer interaction, and communication studies, as it provides valuable information for interpretation, prediction, and decision-making.
 In this paper, we will provide an overview of the concept of context and its categorization, highlighting the importance of context in various fields and applications. We will discuss each type of context and provide examples of how they are used in different fields. Finally, we will conclude by emphasizing the significance of understanding and categorizing context for interpretation, prediction, and decision-making.

An Integrative Engagement Model of Digital Psychotherapy: Exploratory Focus Group Findings.

Digital mental health interventions, such as 2-way and asynchronous messaging therapy, are a growing part of the mental health care treatment ecosystem, yet little is known about how users engage with these interventions over the course of their treatment journeys. User engagement, or client behaviors and therapeutic relationships that facilitate positive treatment outcomes, is a necessary condition for the effectiveness of any digital treatment. Developing a better understanding of the factors that impact user engagement can impact the overall effectiveness of digital psychotherapy. Mapping the user experience in digital therapy may be facilitated by integrating theories from several fields. Specifically, health science's Health Action Process Approach and human-computer interaction's Lived Informatics Model may be usefully synthesized with relational constructs from psychotherapy process-outcome research to identify the determinants of engagement in digital messaging therapy. This study aims to capture insights into digital therapy users' engagement patterns through a qualitative analysis of focus group sessions. We aimed to synthesize emergent intrapersonal and relational determinants of engagement into an integrative framework of engagement in digital therapy. A total of 24 focus group participants were recruited to participate in 1 of 5 synchronous focus group sessions held between October and November 2021. Participant responses were coded by 2 researchers using thematic analysis. Coders identified 10 relevant constructs and 24 subconstructs that can collectively account for users' engagement and experience trajectories in the context of digital therapy. Although users' engagement trajectories in digital therapy varied widely, they were principally informed by intrapsychic factors (eg, self-efficacy and outcome expectancy), interpersonal factors (eg, the therapeutic alliance and its rupture), and external factors (eg, treatment costs and social support). These constructs were organized into a proposed Integrative Engagement Model of Digital Psychotherapy. Notably, every participant in the focus groups indicated that their ability to connect with their therapist was among the most important factors that were considered in continuing or terminating treatment. Engagement in messaging therapy may be usefully approached through an interdisciplinary lens, linking constructs from health science, human-computer interaction studies, and clinical science in an integrative engagement framework. Taken together, our results suggest that users may not view the digital psychotherapy platform itself as a treatment so much as a means of gaining access to a helping provider, that is, users did not see themselves as engaging with a platform but instead viewed their experience as a healing relationship. The findings of this study suggest that a better understanding of user engagement is crucial for enhancing the effectiveness of digital mental health interventions, and future research should continue to explore the underlying factors that contribute to engagement in digital mental health interventions. ClinicalTrials.gov NCT04507360; https://clinicaltrials.gov/ct2/show/NCT04507360.

Metaverse-Based Learning Opportunities and Challenges: A Phenomenological Metaverse Human–Computer Interaction Study

The Metaverse is an end-users-oriented integration of various layers of Information Technology (IT), where Human–Computer Interaction (HCI) would be the core technology. With the rapid development of IT, the Metaverse would allow users to connect, work, conduct business, and access educational resources, all in a technology-mediated environment in new interaction ways. The Metaverse can play a major role in the future of online learning and enable a rich active learning environment, where learners have the opportunity to obtain first-hand experiences that might not be accessible in the physical world. While currently there is a severe shortage in Metaverse-Learning studies, such research strands are expected to soon emerge. The main objective of this paper is to investigate challenges and opportunities for human-centric Metaverse technology in the learning sector, hence accelerating research in this field. A phenomenological research method was used, including semi-structured in-depth interviews, essays written by participants, a focus group discussion with 19 experts in the areas of HCI, intelligent interactive technologies, and online learning. The individual interviews took place in May 2022, with a focus group meeting held online in June 2022 to formulate a collective opinion of the 19 experts. Five challenges were identified for the Metaverse-Learning context: immersive design, privacy and security, universal access, physical and psychological health concerns, and governance. While the research provided suggestions to overcome those challenges, three Meta-Learning opportunities were identified: hands-on training and learning, game-based learning, and collaboration in creating knowledge. The findings of this research contribute to understanding the complexity of the online learning in the Metaverse from the Human–Computer Interaction point of view. These findings can be used to further research the Metaverse as a virtual communication environment and potential business and learning platform.

Reporting non-consensual pornography: clarity, efficiency and distress

According to recent legislative initiatives, non-consensual pornography is a crime in several countries and social media providers have a duty to provide their users easy to use mechanisms to report abuses. In this paper, we analyse the state of the art of the interfaces for reporting non-consensual pornography from the victim’s perspective. Firstly, we analysed 45 content sharing platforms where aggressors might post non-consensual pornography. The analysis identified three distinct interaction styles for reporting the crime: Scriptum (a text-field where the user verbally describes the abuse), Bonam (a multilayered menu that includes a correct option), and Malam (a multilayered menu that does not include a correct option). Secondly, we conducted a within-subject study to evaluate the experience elicited by these interaction styles. Participants (N = 39) were given a scenario and asked to report six blurred images as non-consensual pornography using a medium-fidelity prototype. The results exposed complex trade-offs between clarity, efficiency, and distress among the different interaction styles. These trade-offs open foundational research directions transcending boundaries between human-computer interaction and multimedia studies and interfacing computer science research with the law.

Workers' Health under Algorithmic Management: Emerging Findings and Urgent Research Questions.

Algorithms are increasingly used instead of humans to perform core management functions, yet public health research on the implications of this phenomenon for worker health and well-being has not kept pace with these changing work arrangements. Algorithmic management has the potential to influence several dimensions of job quality with known links to worker health, including workload, income security, task significance, schedule stability, socioemotional rewards, interpersonal relations, decision authority, and organizational trust. To describe the ways algorithmic management may influence workers' health, this review summarizes available literature from public health, sociology, management science, and human-computer interaction studies, highlighting the dimensions of job quality associated with work stress and occupational safety. We focus on the example of work for platform-based food and grocery delivery companies; these businesses are growing rapidly worldwide and their effects on workers and policies to address those effects have received significant attention. We conclude with a discussion of research challenges and needs, with the goal of understanding and addressing the effects of this increasingly used technology on worker health and health equity.

Human-Computer Interaction: Lights, Shadows, and Spotlights

In the last decades, Information and Computer Technology (ICT) has become part and parcel of our lives, bringing undeniable advantages, especially in the workplace, both to individuals and organizations. Nonetheless, the increasingly massive use of ICT also contributed to a higher risk for maladaptive effects on quality of life, thus jeopardizing workers’ well-being. From the perspective of the organizational safety approach, the study of Human-Computer Interaction (HCI) is crucial. Despite this, so far, much attention has been paid to ergonomics and to technostress, but mostly from a sectorial perspective and disregarding several additional phenomena. In this conceptual paper, a multidimensional model able to go beyond the ergonomic features necessary to set a functional physical environment and facilitate the isomorphism mind-ICT is advocated. We point out a modern approach aimed at exploring additional dimensions underpinning HCI in the workplace, with a dual purpose: to highlight possible risk factors for maladjustment and to pave the way for intervention strategies to facilitate a healthy HCI.

A systematic literature review of the Design Critique method

The Design Critique (DC) method is becoming more common in Human Computer Interaction (HCI) and User Experience (UX) studies as the need for new evaluation methods of emerging technologies is increasing. However, there is an clear lack of guidelines on how to conduct DC studies in the UX context. The goal of this paper is to provide an overview of the DC method in the fields of UX. In addition, this paper aims to propose a generic process of running DC studies in the same context. We present a systematic literature review of the DC method. Moreover, we conduct a course of thematic analysis on the selected papers to identify the various DC processes and explore the following attributes: participant categories, data collection methods, and data analysis methods in each process. We identified three different trends of DC processes: detailed, moderate and minimal. In addition, we proposed a generic DC process consisting of 10 steps divided into three main phases: preparation, conducting design critique, and pro-processing. We found that domain experts represent the majority of studies participants. Using interviews to collect qualitative data and using script coding analysis are the two most common methods of collecting and analyzing data. Conducting DC studies can improve overall systems usability by addressing design flaws at an early stage of development. The process of conducting a DC varies, depending on the project goals and states. The DC method aligns well with the small light-weight steps approach in Agile methods.

Towards Context-Aware Facial Emotion Reaction Database for Dyadic Interaction Settings

Emotion recognition is a significant issue in many sectors that use human emotion reactions as communication for marketing, technological equipment, or human-robot interaction. The realistic facial behavior of social robots and artificial agents is still a challenge, limiting their emotional credibility in dyadic face-to-face situations with humans. One obstacle is the lack of appropriate training data on how humans typically interact in such settings. This article focused on collecting the facial behavior of 60 participants to create a new type of dyadic emotion reaction database. For this purpose, we propose a methodology that automatically captures the facial expressions of participants via webcam while they are engaged with other people (facial videos) in emotionally primed contexts. The data were then analyzed using three different Facial Expression Analysis (FEA) tools: iMotions, the Mini-Xception model, and the Py-Feat FEA toolkit. Although the emotion reactions were reported as genuine, the comparative analysis between the aforementioned models could not agree with a single emotion reaction prediction. Based on this result, a more-robust and -effective model for emotion reaction prediction is needed. The relevance of this work for human-computer interaction studies lies in its novel approach to developing adaptive behaviors for synthetic human-like beings (virtual or robotic), allowing them to simulate human facial interaction behavior in contextually varying dyadic situations with humans. This article should be useful for researchers using human emotion analysis while deciding on a suitable methodology to collect facial expression reactions in a dyadic setting.

Reliability of MUSE 2 and Tobii Pro Nano at capturing mobile application users' real-time cognitive workload changes.

Despite the importance of cognitive workload in examining the usability of smartphone applications and the popularity of smartphone usage globally, cognitive workload as one attribute of usability tends to be overlooked in Human-Computer Interaction (HCI) studies. Moreover, limited studies that have examined the cognitive workload aspect often measured some summative workloads using subjective measures (e.g., questionnaires). A significant limitation of subjective measures is that they can only assess the overall, subject-perceived cognitive workload after the procedures/tasks have been completed. Such measurements do not reflect the real-time workload fluctuation during the procedures. The reliability of some devices on a smartphone setting has not been thoroughly evaluated. This study used mixed methods to empirically study the reliability of an eye-tracking device (i.e., Tobii Pro Nano) and a low-cost electroencephalogram (EEG) device (i.e., MUSE 2) for detecting real-time cognitive workload changes during N-back tasks. Results suggest that the EEG measurements collected by MUSE 2 are not very useful as indicators of cognitive workload changes in our setting, eye movement measurements collected by Tobii Pro Nano with mobile testing accessory are useful for monitoring cognitive workload fluctuations and tracking down interface design issues in a smartphone setting, and more specifically, the maximum pupil diameter is the preeminent indicator of cognitive workload surges. In conclusion, the pupil diameter measure combined with other subjective ratings would provide a comprehensive user experience assessment of mobile applications. They can also be used to verify the successfulness of a user interface design solution in improving user experience.

Social-Spiritual Face: Designing Social Reading Support for Spiritual Well-being

Techno-spiritual practices refer to the use of digital technologies to support various spiritual activities, such as scripture reading. While prior human-computer interaction studies largely focus on understanding techno-spiritual practices in both personal and ministry environments, there is a lack of design research that explores novel design opportunities, based on longitudinal field deployment. As an important techno-spiritual practice, this work focuses on scripture reading and investigates the design space of "social scripture reading," as it is often organized into small groups for successful behavior maintenance. We designed and evaluated BibleCell, a social scripture reading tool that supports personalized reading plans, scripture reading, and social sharing. After the third year of deployment, we performed a two-month user study in Korean Protestant churches to deepen our understanding of techno-spiritual practices in social contexts via in-depth interviews (n = 27). We report the major themes of social techno-spiritual practices, such as social motivators, social interaction patterns, and leadership roles. We discuss our findings using a novel design concept of social-spiritual awareness that considers both the social and spiritual aspects of interactions in social computing systems.

3D Skeletal Volume Templates for Deep Learning-Based Activity Recognition

Due to advances in depth sensor technologies, the use of these sensors has positively impacted studies of human-computer interaction and activity recognition. This study proposes a novel 3D action template generated from depth sequence data and two methods to classify single-person activities using this 3D template. Initially, joint skeleton-based three-dimensional volumetric templates are constructed from depth information. In the first method, images are obtained from various view angles of these three-dimensional templates and used for deep feature extraction using a pre-trained convolutional neural network. In our experiments, a pre-trained AlexNet model trained with the ImageNet dataset is used as a feature extractor. Activities are classified by combining deep features and Histogram of Oriented Gradient (HOG) features. The second approach proposes a three-dimensional convolutional neural network that uses volumetric templates as input for activity classification. Proposed methods have been tested with two publicly available datasets. Experiments provided promising results compared with the other studies presented in the literature.