Human-computer Interaction Evaluation Research Articles

Who Made That Decision and Why? Users’ Perceptions of Human Versus AI Decision-Making and the Power of Explainable-AI

With the advent of artificial intelligence (AI) based systems, a new era has begun. Decisions that were once made by humans are now increasingly being made by these advanced systems, with the inevitable consequence of our growing reliance on AI in many aspects of our lives. At the same time, the opaque nature of AI-based systems and the possibility of unintentional or hidden discriminatory practices and biases raises profound questions not only about the mechanics of AI, but also about how users perceive the fairness of these systems. We hypothesize that providing various explanations for AI decision-making processes and output may enhance users’ fairness perceptions and make them trust the system and adopt its decisions. Hence, we devised an online between-subject experiment that explores users’ fairness and comprehension perceptions of AI systems with respect to the explanations provided by the system, employing a case study of a managerial decision in the human resources (HR) domain. We manipulated (i) the decision-maker (AI or human); (ii) the input (candidate characteristics); (iii) the output (recommendation valence), and (iv) the explanation style. We examined the effect of the various manipulations (and individuals’ demographic and personality characteristics) using multivariate ordinal regression. We also performed a multi-level analysis of experiment components to examine the effects of the decision-maker type, explanation style, and their combination. The results suggest three main conclusions. The first conclusion is that there is a gap in users’ fairness and comprehension perception of AI-based decision making systems compared to human decision making. The second conclusion is that knowing that an AI-based system provided the decisions negatively affects users’ fairness and comprehension perceptions, compared to knowing that humans made the decision. Finally, the third conclusion is that providing case-based, certification-based, or sensitivity-based explanations can narrow this gap and may even eliminate it. Additionally, we found that users’ fairness and comprehension perceptions are influenced by a variety of factors such as the input, output, and explanation provided by the system, as well as by individuals’ age, education, computer skills, and personality. Our findings may help to understand when and how to use explanations to improve users’ perceptions regarding AI-based decision-making. CCS CONCEPTS • Human computer interaction (HCI) → HCI design and evaluation methods → User studies • Human-centered computing → Human computer interaction (HCI) → Empirical studies in HCI • Applied computing → Law, social and behavioral sciences → Sociology

Identifying the Importance of UX Dimensions for Different Software Product Categories

Abstract Billions of users around the world use mobile applications and computer software to achieve their professional and personal goals. This situation drives User Experience (UX) researchers and practitioners to assess the importance of UX dimensions across different products, to facilitate the design, development and evaluation of new products. To that end, this study surveyed a group of 200 end users and 8 UX experts from Canada to document the importance of 21 UX dimensions for 15 software product categories. The results confirmed that the importance of UX dimensions varies between product categories. Comparing the findings to those of similar studies conducted in Germany and Indonesia revealed that, while culture influences the rating of UX dimensions, the importance of UX dimensions is still determined by the product category. Comparisons between the importance ratings of UX dimensions between end users and experts and within end users were not significant in 77% and 97% of cases, respectively. Results showed that task-based product categories rely more on pragmatic dimensions (i.e. functionality and usability) while leisure-based products value hedonic dimensions (i.e. pleasure) as well. This study benefits researchers and practitioners by enabling them to select the most important UX dimensions for evaluating their products. CCS CONCEPTS: • Human-centered computing • Human-Computer Interaction (HCI) • HCI design and evaluation methods. Additional Keywords and Phrases: User experience, UX dimension, UX evaluation, culture.

Interactive Evolutionary Design of Handbag Integrating Bamboo Weaving Material

Abstract The integration and development of fashion products and intangible cultural heritage have gradually become the highlight of product design. Following thousands of years of evolution, bamboo weaving patterns offer a rich resource library for use by designers. However, the designers cannot predict which bamboo weaving pattern can be adopted into women’s handbags to satisfy consumers’ aesthetics. Therefore, taking the form design of women’s handbags as an example, this study develops an artificial intelligence-based program through which consumers can codesign with designers to effectively match the bamboo weaving pattern with the design style of women’s handbags. First, based on the coding, selection, and mutation processes of interactive genetic algorithms, consumer preferences, and perception evaluations are effectively incorporated into product design. This ensures that the evolved product solutions meet consumer preferences. Second, considering evaluator fatigue in human–computer interaction evaluation, the decision tree algorithm is used as a proxy for evaluation. Finally, the developed interactive system supports mass-generated bamboo-woven bags of different styles, allowing prediction of consumers’ preferences for the design of women’s handbags. Furthermore, the system can assist designers and bamboo craftsmen in designing other bamboo or wooden products.

A Hybrid Multimodal Emotion Recognition Framework for UX Evaluation Using Generalized Mixture Functions.

Multimodal emotion recognition has gained much traction in the field of affective computing, human-computer interaction (HCI), artificial intelligence (AI), and user experience (UX). There is growing demand to automate analysis of user emotion towards HCI, AI, and UX evaluation applications for providing affective services. Emotions are increasingly being used, obtained through the videos, audio, text or physiological signals. This has led to process emotions from multiple modalities, usually combined through ensemble-based systems with static weights. Due to numerous limitations like missing modality data, inter-class variations, and intra-class similarities, an effective weighting scheme is thus required to improve the aforementioned discrimination between modalities. This article takes into account the importance of difference between multiple modalities and assigns dynamic weights to them by adapting a more efficient combination process with the application of generalized mixture (GM) functions. Therefore, we present a hybrid multimodal emotion recognition (H-MMER) framework using multi-view learning approach for unimodal emotion recognition and introducing multimodal feature fusion level, and decision level fusion using GM functions. In an experimental study, we evaluated the ability of our proposed framework to model a set of four different emotional states (Happiness, Neutral, Sadness, and Anger) and found that most of them can be modeled well with significantly high accuracy using GM functions. The experiment shows that the proposed framework can model emotional states with an average accuracy of 98.19% and indicates significant gain in terms of performance in contrast to traditional approaches. The overall evaluation results indicate that we can identify emotional states with high accuracy and increase the robustness of an emotion classification system required for UX measurement.

A review of cyber vigilance tasks for network defense.

Human-centered computing~Human computer interaction (HCI)~HCI design and evaluation methods.

Technology Acceptance Model: An Analysis of Human-Computer Interaction in The UASBN Educational Game

The Ujian Akhir Sekolah Berstandar Nasional (UASBN) educational game is designed as a means of training for students to face school final exams in 6th grade. Before implementing this educational game, it must be tested first related to user acceptance of the technology created or from the human-computer interaction (HCI) side. A total of 173 students tested this educational game, and then data was collected based on the criteria from the Technology Acceptance Model (TAM). The measurement criteria include Perceived Usefulness (TPU) and Perceived Ease of Use (TPE) variables, each of which has aspects of functionality (TFL), accessibility (TAC), and Learning Goal Orientation (TLGO). In contrast, the dependent variable is Behaviour Intention (TBI). The SEM (AMOS) and SPSS statistical packages were used to analyze this HCI evaluation study. A questionnaire with 21 questions was used to collect the necessary data for analysis. The findings of the estimation demonstrate that each component of TAM contributes pretty significantly. In total, user acceptance of the UASBN educational game is a TPE variable contributing 39% to the UASBN human-computer interaction game. Meanwhile, the TPU variable and its variants contributed 73% to the human-computer interaction game UASBN, and Meanwhile, TBI gave a role of 50.4% to the human-computer interaction game UASBN education. This shows that human-computer interaction with educational games through the technology acceptance model criteria contributed 50.4%, while 49.6% was contributed by other variables not raised in this research. This demonstrates that the educational requirements for student gaming training can be satisfactorily met by users and produce favorable outcomes.
 

Temporal Resolution Capabilities of the Mid-Frequency Heart Rate Variability-based Human-Computer Interaction Evaluation Method

Temporal Resolution Capabilities of the Mid-Frequency Heart Rate Variability-based Human-Computer Interaction Evaluation Method

A Multi-National Human–Computer Interaction Evaluation of the Public Participatory GIS GeoCitizen

Designing user-friendly Public Participatory Geographic Information Systems (PPGIS) is a challenging task, since a PPGIS is typically used by users who have different characteristics and different requirements and needs. Hence, applying Human–Computer Interaction (HCI) principles is of particular importance in designing PPGIS. This study aims to create an inventory of usability aspects of a PPGIS by focusing on understanding the characteristics of a broad range of users. The usability study included 73 participants from Colombia, Uganda and Austria. We combined a custom qualitative survey (conducted in all three countries) with an eye-tracking based survey (conducted only in Austria). Considerable usability problems were faced especially by participants with low levels of IT-literacy. This was mostly due to a lack of experience in using functionally complex smartphone applications or interactive maps. In general, we observed a high level of difference in usability between the user groups. The eye-tracking statistics for the Austrian study supported the outcomes of the qualitative survey well.

AQUArIUM - A suite of software measures for HCI quality evaluation of ubiquitous mobile applications

Five characteristics for HCI evaluation in ubiquitous applications are selected.Twenty-four quality measures to evaluate these five characteristics are proposed.A theoretical and empirical validations were performed. Ubiquitous computing has changed the way users interact with technology. Its applications are everywhere, supporting users in everyday activities in a transparent way with little or no need for attention. To ensure the adoption of these applications, it is essential to assess the quality of the interaction with its users. To do that, measurements can be applied to obtain data about quality characteristics in a software product. In a previous study about quality characteristics and software measures for ubiquitous computing, we have identified a gap in the literature regarding software measures for evaluating essential quality characteristics in ubiquitous systems. Therefore, this paper proposes a suite of well-defined software measures (twenty-four in total), called AQUArIUM, to evaluate human-computer interaction (HCI) in ubiquitous applications for mobile devices. These measures address five quality characteristics of ubiquitous computing: Context-awareness, Mobility, Attention, Calmness and Transparency. The proposed suite was validated theoretically and empirically. The results showed not only that the suite is feasible, but also that it indicates specific problems of HCI quality, which are helpful in improving the evaluated ubiquitous mobile application.

Quality characteristics and measures for human–computer interaction evaluation in ubiquitous systems

The advent of ubiquitous systems places even more focus on users, since these systems must support their daily activities in such a transparent way that does not disturb them. Thus, much more attention should be provided to human---computer interaction (HCI) and, as a consequence, to its quality. Dealing with quality issues implies first the identification of the quality characteristics that should be achieved and, then, which software measures should be used to evaluate them in a target system. Therefore, this work aims to identify what quality characteristics and measures have been used for the HCI evaluation of ubiquitous systems. In order to achieve our goal, we performed a large literature review, using a systematic mapping study, and we present our results in this paper. We identified 41 pertinent papers that were deeply analyzed to extract quality characteristics and software measures. We found 186 quality characteristics, but since there were divergences on their definitions and duplicated characteristics, an analysis of synonyms by peer review based on the equivalence of definitions was also done. This analysis allowed us to define a final suitable set composed of 27 quality characteristics, where 21 are generic to any system but are particularized for ubiquitous applications and 6 are specific for this domain. We also found 218 citations of measures associated with the characteristics, although the majority of them are simple definitions with no detail about their measurement functions. Our results provide not only an overview of this area to guide researchers in directing their efforts but also it can help practitioners in evaluating ubiquitous systems using these measures.

Double Binds and Double Blinds: Evaluation Tactics in Critically Oriented HCI

<div class="page" title="Page 1"><div class="layoutArea"><div class="column"><p><span>Critically oriented researchers within Human-Computer Interaction (HCI) have fruitfully intersected design and critical analysis to engage users and designers in reflection on underlying values, assumptions and dominant practices in technology. To successfully integrate this work within the HCI community, critically oriented researchers have tactically engaged with dominant practices within HCI in the design and evaluation of their work. This paper draws attention to the ways that tactical engagement with aspects of HCI evaluation methodology shapes and bears consequences for critically oriented research. We reflect on three of our own experiences evaluating critically oriented designs and trace challenges that we faced to the ways that sensibilities about generalizable knowledge are manifested in HCI evaluation methodology. Drawing from our own experiences, as well as other influential critically oriented design projects in HCI, we articulate some of the trade-offs involved in consciously adopting or not adopting certain normative aspects of HCI evaluation. We argue that some forms of this engagement can hamstring researchers from pursuing their intended research goals and have consequences beyond specific research projects to affect the normative discourse in the field as a whole.</span></p></div></div></div>

REVIEW DESAIN INTERFACE APLIKASI SOPPPOS MENGGUNAKAN EVALUASI HEURISTIK

ABSTRAK Tujuan dari artikel ini adalah melakukan sebuah review terhadap desain antar muka (interface) pada aplikasi yang bernama System Online Payment Point PT Pos Indonesia atau disingkat SOPPPOS. Review menggunakan evaluasi interaksi manusia komputer yang bernama evaluasi heuristik yang diperkenalkan oleh Molich dan Nielsen. Berdasarkan hasil evaluasi menyatakan bahwa secara umum desain interface dan kemudahan pengguna (usability) dari aplikasi SOPPPOS cukup baik, namun masih ada beberapa hal yang perlu diperbaiki terutama dalam karakteristik kemudahan penggunaan. Kata kunci: review, interface, aplikasi SOPPPOS, evaluasi, heuristik.

Redesign of a computerized clinical reminder for colorectal cancer screening: a human-computer interaction evaluation

BackgroundBased on barriers to the use of computerized clinical decision support (CDS) learned in an earlier field study, we prototyped design enhancements to the Veterans Health Administration's (VHA's) colorectal cancer (CRC) screening clinical reminder to compare against the VHA's current CRC reminder.MethodsIn a controlled simulation experiment, 12 primary care providers (PCPs) used prototypes of the current and redesigned CRC screening reminder in a within-subject comparison. Quantitative measurements were based on a usability survey, workload assessment instrument, and workflow integration survey. We also collected qualitative data on both designs.ResultsDesign enhancements to the VHA's existing CRC screening clinical reminder positively impacted aspects of usability and workflow integration but not workload. The qualitative analysis revealed broad support across participants for the design enhancements with specific suggestions for improving the reminder further.ConclusionsThis study demonstrates the value of a human-computer interaction evaluation in informing the redesign of information tools to foster uptake, integration into workflow, and use in clinical practice.

An Empirical Study of Relationships Between Traditional Usability

Task performance data and subjective assessment data are widely used as usability measures in the human-computer interaction (HCI) field. Recently, physiology has also been explored as a metric for evaluating usability. However, it is not clear how physiological measures relate to traditional usability evaluation indexes. In this paper, we investigate the relationships among three kinds of data: task performance, subjective assessment and physiological measures. We found evidence that physiological data correlate with task performance data in a video game: with a decrease of the task performance level, the normalized galvanic skin response (GSR) increases. In addition, physiological data are mirrored in subjective reports assessing stress level. The research provides an initial step toward using physiology as a complementary or an independent usability measure for HCI evaluation.

Enterprise Resource Planning Interactive Via duct B/w Human & Computer

Understanding human thinking is crucial in the design and evaluation of human-computer interaction. Computing devices and applications are at this moment employed ahead of the desktop, in dissimilar environments, and this tendency toward ubiquitous computing is gathering speed. As computers become major necessity and having connectivity widespread, we are increasingly becoming competent to access computer power, data, information and knowledge from anyplace and at anytime. Conversely, in order to fetch the benefits of such accessible intelligence (commonly known as ‘ERP’ or a latest buzzword in a contemporary scenario and in the field of information technology too); we should not overlook the ongoing evolutions and revolutions in human-computer communications and its interfaces. Indeed, as the human factors of information systems and knowledge systems emerge as a research area in and of itself, it is thoughtful to go on board on this electrifying field of study. The operation of the Human-Computer with a flavour of ERP is to encourage interdisciplinary study and education in user-centered computer systems. The location of ERP is to make interactive and intelligent human-computer interfaces, in order to effectively enable users to accomplish their desired tasks. This paper present a personal outlook of the HCI landscape in a historical perspective. The paper also aims part to support newcomers in the filed to grasp the origins of HCI and in part to provide grounds for a discussion of the field of usability that is being challenged by the social and cultural developments (Jorgensen 2000). This paper has argued that in order to properly understand the interaction between ERP system and human computer interactions networks one must scrutinize the mutual flows of influence and the dynamic interaction between the two. Thus to bridge this gap, the paper critically reviews the existing ERP in a humanity context and upgrade decision-drivers, synthesizes a framework based on the literature, and extends the framework as necessary. At last but not least the paper show-off a personal, historical overview of these develoments in accompanying ERP system as seen form a HCI perspective.

A simulation-based embedded probe technique for human-computer interaction evaluation

There is growing interest in developing video-based methods to evaluate the usefulness and usability of computerised tools in healthcare. In this paper, we propose a human-computer interaction evaluation method that protects the confidentiality of patient information and reduces litigation risks for participants by embedding and analysing performance on "probes" in complex scenarios with high face validity in a simulated setting. We describe the application of the method to a series of three studies of bar code medication administration (BCMA) software, used to reduce medication errors in the Veteran's Administration.

The Effect of Affect: The Hedonomic Evaluation of Human-Computer Interaction

We evaluated the use of color in a human-computer interface to investigate how affective cues influence usability judgments. Yellow and green, were found to enhance performance on a cognitive flexibility task as compared to white and gray. Such an outcome suggests that color promotes positive affect. The aesthetically designed interface that promoted positive affect was also found to enhance perceptions of interface usability. This relationship, however, was only found when underlying usability was effective. When usability was poor, aesthetics made users more cognizant of the usability weaknesses. The results suggest that designers consider incorporating color into designs to help elicit positive emotion in users but that they must be aware that incorporation of aesthetics comes with the potential cost of disposing users to be more discerning of usability.

A Hierarchical Task Model for Dispatching in Computer-Assisted Demand-Responsive Paratransit Operation

This research was conducted to organize and model a comprehensive set of paratransit dispatching task activity in a computer-assisted control room. The dispatching task involves continuous monitoring and interaction with archived and real-time data using a typical dispatch software. Extensive field observations and video recordings of dispatch operations were conducted. Then, a skeleton decision model of the dispatchers activities was generated and later validated with an expert dispatcher. Hierarchical Task Analysis (HTA) was used to further refine this model into a comprehensive HTA decision hierarchy (decision tree). To capture the complexity of multi-dispatcher operation, we extended this model to a simultaneous operation of two collaborative dispatchers in the same control room—a van dispatcher and a lead dispatcher. The results of this analysis showed that HTA could be used to model such an activity with a high degree of fidelity. The sequential decision tree format of the HTA also shows promise as a training tool for entry-level dispatch personnel and as a tool for human-computer interaction evaluation in dispatch software design.

Video analysis in cognitive ergonomics: a methodological perspective.

The object of this paper is to discuss the methodological issues associated with using video analysis in the evaluation of Human-Computer Interaction. It is proposed that the mapping between low-level observed events and the high-level goals of the usability evaluation ultimately determines the success of this research technique. The paper firstly addresses the advantages of using video as a data-capture medium in the context of usability evaluations, and secondly, evaluates the use of alternative data capture devices. The process of conducting video analysis is then described. The methodological issues that arise in employing this technique are presented in a format consistent with the execution of any empirical investigation: research design, scoring, data analysis and interpretation of results. The reader is alerted to certain methodological problems associated with the use of video analysis as a means of data collection. The conclusion is drawn that video analysis provides a very powerful data source for the study of human performance in cognitive ergonomics and usability evaluations provided that human factors practitioners attend to the methodological requirements imposed by this technique.

The reaction of psychiatric patients to computerized assessment.

A questionnaire was given following test administration in either a standard or computerized form, to 367 psychiatric patients who completed up to eight psychometric tests. Patients responded in the computerized versions by using either a keyboard or a touch-screen. In general, the evaluation of both test forms was positive and fewer differences were detected than had been expected. Where differences existed, the computerized forms were found more enjoyable, especially in the case of the simpler tests, but were also reported to be less comprehensible and less clear in the case of some of the more complex ones. Subjective estimates of attainment and willingness to undergo further testing were largely unaffected by test form. The touch-screen was preferred to the keyboard in the simpler tests, but in other cases there was evidence that prior experience of the standard form of the test facilitated performance. The findings support the feasibility of creating computerized tests which are essentially parallel to standard original forms, but point to the need to understand the complex factors which govern the users' evaluation of human-computer interaction. However, computerized assessment seems to be entirely acceptable to clinical psychiatric patients.