Human-computer Interaction Research Research Articles

Combinatorial Optimization of Graphical User Interface Designs

The graphical user interface (GUI) has become the prime means for interacting with computing systems. It leverages human perceptual and motor capabilities for elementary tasks such as command exploration and invocation, information search, and multitasking. For designing a GUI, numerous interconnected decisions must be made such that the outcome strikes a balance between human factors and technical objectives. Normally, design choices are specified manually and coded within the software by professional designers and developers. This article surveys combinatorial optimization as a flexible and powerful tool for computational generation and adaptation of GUIs. As recently as 15 years ago, applications were limited to keyboards and widget layouts. The obstacle has been the mathematical definition of design tasks, on the one hand, and the lack of objective functions that capture essential aspects of human behavior, on the other. This article presents definitions of layout design problems as integer programming tasks, a coherent formalism that permits identification of problem types, analysis of their complexity, and exploitation of known algorithmic solutions. It then surveys advances in formulating evaluative functions for common design-goal foci such as user performance and experience. The convergence of these two advances has expanded the range of solvable problems. Approaches to practical deployment are outlined with a wide spectrum of applications. This article concludes by discussing the position of this application area within optimization and human-computer interaction research and outlines challenges for future work.

Modeling interaction as a complex system

ABSTRACT Researchers in Human-Computer Interaction typically rely on experiments to assess the causal effects of experimental conditions on variables of interest. Although this classic approach can be very useful, it offers little help in tackling questions of causality in the kind of data that are increasingly common in HCI – capturing user behavior ‘in the wild.’ To analyze such data, model-based regressions such as cross-lagged panel models or vector autoregressions can be used, but these require parametric assumptions about the structural form of effects among the variables. To overcome some of the limitations associated with experiments and model-based regressions, we adopt and extend ‘empirical dynamic modelling’ methods from ecology that lend themselves to conceptualizing multiple users’ behavior as complex nonlinear dynamical systems. Extending a method known as ‘convergent cross mapping’ or CCM, we show how to make causal inferences that do not rely on experimental manipulations or model-based regressions and, by virtue of being non-parametric, can accommodate data emanating from complex nonlinear dynamical systems. By using this approach for multiple users, which we call ‘multiple convergent cross mapping’ or MCCM, researchers can achieve a better understanding of the interactions between users and technology – by distinguishing causality from correlation – in real-world settings.

Investigating Agreement Among HCI Researchers About Human–Computer Interaction in CMMI-DEV Model: A Case Study

AbstractDo human–computer interaction (HCI) researchers agree about which HCI approaches should be used in the development of an interactive system? To address this question, we performed a study based on the engineering process areas (requirements development, technical solution, product integration, verification and validation) of the CMMI-DEV (Capability Maturity Model Integration for Development), a software engineering model commonly used in the industry. To carry out this study, a literature review was performed, a specific instrument was designed based on CMMI-DEV and interviews with 20 researchers from the HCI domain were conducted. Analyzing the interview data with monovariate and multivariate statistical approach (Multiple Correspondence Analysis - MCA), we find the greatest agreement occurred among researchers for HCI approaches that support verification and validation phases and the use of functional prototypes in some activities of the last phases of software development (technical solution and product integration). However, we identified lack of agreement among researchers regarding requirements development activities.

Human accuracy in mobile data collection

The collection of participant data ‘in the wild’ is widely employed by Human-Computer Interaction researchers. A variety of methods, including experience sampling, mobile crowdsourcing, and citizen science, rely on repeated participant contributions for data collection. Given this strong reliance on participant data, ensuring that the data is complete, reliable, timely, and accurate is key. Although previous work has made significant progress on ensuring that a sufficient amount of data is collected, the accuracy of human contributions has remained underexposed. In this article we argue for an emerging need for an increased focus on this aspect of human-labelled data. The articles published in this special issue demonstrate how a focus on the accuracy of the collected data has implications on all aspects of a study – ranging from study design to the analysis and reporting of results. We put forward a five-point research agenda in which we outline future opportunities in assessing and improving human accuracy in mobile data collection.

Exploring the design of avatars for users from Arabian culture through a hybrid approach of deductive and inductive reasoning

While avatars are frequently employed as a user interface (UI) element for improving user experience in human-computer interaction, the current design of avatars is primarily dominated by non-Arabian cultures. To the best of our knowledge, no previous research or guidelines (based on empirical evidence) can be found for avatar design in the context of Arabian culture. Aiming to address this gap, we conducted an exploratory study to investigate how avatars can be designed for Arab users. Following a hybrid approach of deductive and inductive reasoning, we reviewed the literature on UI design for Arabian culture, (non-Arabian) avatar design in human-computer interaction research, and social response theory. Subsequently, we conducted 32 semi-structured interviews with Arabian culture experts, psychologists, and potential users. Based on thematic analysis of the interviews, we developed a set of six general guidelines for the design of avatars for users from Arabian culture. These six guidelines are expected to provide system or UI designers the ability to design and employ appropriate avatars that can promote Arab users’ experience and engagement.

Towards the Detection of UX Smells: The Support of Visualizations

Daily experiences in working with various types of computer systems show that, despite the offered functionalities, users have many difficulties, which affect their overall User eXperience (UX). The UX focus is on aesthetics, emotions and social involvement, but usability has a great influence on UX. Usability evaluation is acknowledged as a fundamental activity of the entire development process in software practices. Research in Human-Computer Interaction has proposed methods and tools to support usability evaluation. However, when performing an evaluation study, novice evaluators still have difficulties to identify usability problems and to understand their causes: they would need easier to use and possibly automated tools. This article describes four visualization techniques whose aim is to support the work of evaluators when performing usability tests to evaluate websites. Specifically, they help detect “usability smells”, i.e. hints on web pages that might present usability problems, by visualizing the paths followed by the test participants when navigating in a website to perform a test task. A user study with 15 participants compared the four techniques and revealed that the proposed visualizations have the potential to be valuable tools for novice usability evaluators. These first results should push researchers towards the development of further tools that are capable to support the detection of other types of UX smells in the evaluation of computer systems and that can be translated into common industry practices.

Do We Truly Sacrifice Truth for Simplicity: Comparing Complete Individual Randomization and Semi- Randomized Approaches to Survey Administration

Human-computer interaction researchers have long used survey methodologies. However, debate remains about the potential for participants to provide biased responses to subsequent items based on previously viewed items. In this research, we investigate the effects of survey item ordering that researchers have not studied previously. Grounded in previous exploratory item-ordering studies using an HCI online survey, we investigate bias in more detail. In addition, we use an adult sample population so that we can extend our results more broadly as compared to previous research. We employed two distinct randomizing survey approaches: 1) complete item randomization for each respondent (random), which presents items to each respondent in a completely randomized order; and 2) partially individualized item randomization (grouped), which presents constructs in the same order in a survey but randomizes items in each construct for each respondent. Our results suggest researchers should use fully randomized survey instruments in HCI research whenever possible since grouped ordering of any kind increases bias and statistical inflation, which can influence results’ veracity. Additionally, we did not appear to find any significant increase in the participants’ frustration or fatigue to be associated with the random treatment.

Multimodal data fusion algorithm applied to robots

In recent years, the use of multimodal human-computer interaction technology to achieve the enhancement of human intelligence has become a new topic in human-computer interaction research. When the robot can’t react correctly in a single mode, it is necessary to realize multimodal fusion. To this end, this paper proposes a multimodal fusion algorithm that applies the data obtained by the CNN feature layer to the decision-level. The speech recognition text is semantically matched with the text in the text library, and the similar probability vector is returned. At the same time, the similarity probability vector of the gesture recognition is obtained, and the data is filtered by the threshold, and the set of high probability data codes is assigned to the two modes. The intersection operation, and the final instruction is sent to the robot. The experimental results show that the influence of environmental factors on the single channel result is reduced, and the single mode ambiguity problem is eliminated. The multi-channel fusion algorithm with additional weight is more accurate than the common multi-channel fusion algorithm. At the same time, it has also been well received by many test users.

Potential Eye Tracking Metrics and Indicators to Measure Cognitive Load in Human-Computer Interaction Research

Potential Eye Tracking Metrics and Indicators to Measure Cognitive Load in Human-Computer Interaction Research

GFocus: User Focus-based Graph Query Autocompletion

Graph query autocompletion ( <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">gQAC</small> ) generates a small list of ranked query suggestions during the graph query formulation process in a visual environment. The current state-of-the-art of <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">gQAC</small> provides suggestions that are formed by adding subgraph increments to arbitrary places of an existing (partial) user query. However, according to the research results on human-computer interaction (HCI), humans can only interact with a small number of recent software artifacts in hand. Hence, many of such suggestions could be irrelevant. In this paper, we present the <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">GFocus</small> framework that exploits a novel notion of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">user focus of graph query formulation</i> (or simply <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">focus</i> ). Intuitively, the focus is the subgraph that a user is working on. We formulate <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">locality principles</i> inspired by the HCI research to automatically identify and maintain the focus. We propose novel monotone submodular ranking functions for generating <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">popular</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">comprehensive</i> query suggestions only at the focus. In particular, the query suggestions of <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">GFocus</small> have high result counts (when they are used as queries) and maximally cover the possible suggestions at the focus. We propose efficient algorithms and an index for ranking the suggestions. Our results show that <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">GFocus</small> saves 12-32 percent more mouse clicks and is 35× more efficient than the state-of-the-art competitor.

A Computationally Efficient sEMG based Silent Speech Interface using Channel Reduction and Decision Tree based Classification

Abstract Silent Speech Interface is one of the promising areas of Human-Computer Interaction research. The surface electromyography based silent speech interface is a technique where the electric activity of facial muscles are used to detect speech. The existing sEMG based SSI techniques use complex machine learning algorithms and too many number of electrodes on the subject’s face. It creates inconvenience to the user who might have undergone laryngectomy. More number of electrodes becomes highly invasive to the user, while complex classification algorithms increase the computational cost and prevents real time implementation of sEMG based SSI. Thus the objective of this research work was to develop a less complex and computationally less expensive model to classify words. To achieve this goal channel reduction technique and the use of Decision Tree based classification algorithm was employed. Only the time domain features are used as input to the classification algorithm. The motive was to exploit the advantage of computational ease in extracting the time domain features as compared to the frequency domain features. The sEMG data of the words used in this work are obtained from the complete utterance of the sentences and not by individual utterances of the word. Our algorithm was able to achieve a word accuracy of 95.17% even after applying a channel reduction, thereby allowing us to use only the data of 5 channels, in place of a conventional seven channel setup.

A CNN-Assisted Enhanced Audio Signal Processing for Speech Emotion Recognition.

Speech is the most significant mode of communication among human beings and a potential method for human-computer interaction (HCI) by using a microphone sensor. Quantifiable emotion recognition using these sensors from speech signals is an emerging area of research in HCI, which applies to multiple applications such as human-reboot interaction, virtual reality, behavior assessment, healthcare, and emergency call centers to determine the speaker’s emotional state from an individual’s speech. In this paper, we present major contributions for; (i) increasing the accuracy of speech emotion recognition (SER) compared to state of the art and (ii) reducing the computational complexity of the presented SER model. We propose an artificial intelligence-assisted deep stride convolutional neural network (DSCNN) architecture using the plain nets strategy to learn salient and discriminative features from spectrogram of speech signals that are enhanced in prior steps to perform better. Local hidden patterns are learned in convolutional layers with special strides to down-sample the feature maps rather than pooling layer and global discriminative features are learned in fully connected layers. A SoftMax classifier is used for the classification of emotions in speech. The proposed technique is evaluated on Interactive Emotional Dyadic Motion Capture (IEMOCAP) and Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) datasets to improve accuracy by 7.85% and 4.5%, respectively, with the model size reduced by 34.5 MB. It proves the effectiveness and significance of the proposed SER technique and reveals its applicability in real-world applications.

A comparative assessment of Wi-Fi and acoustic signal-based HCI methods on the practicality

Wi-Fi and acoustic signal-based human–computer interaction (HCI) methods have received growing attention in academia. However, there still are issues to be addressed despite their flourishing. In this work, we evaluate the practicality of the state-of-the-art signal-based HCI research in terms of the following six aspects—granularity, robustness, usability, efficiency, stability, and deployability. The paper presents our analysis results, observations and prospective research directions. We believe that this work will serve as a standard for future signal-based HCI research for assessing the practicality of newly developed methods.

Engaging Feminist Solidarity for Comparative Research, Design, and Practice

Research in the fields of Computer Supported Cooperative Work (CSCW) and Human-Computer Interaction (HCI) is increasingly embracing and moving across borders. While universalism in such research is widely rejected, sole focus on the "particular'' is also commonly critiqued. Kentaro Toyama unpacks this tension, calling for balance via "deliberate efforts to understand the interplay of human universals and cultural differences, and how it pertains to design.'' In this paper, one such deliberate effort, we introduce the notion of feminist solidarity---as theorized by Chandra Mohanty---for drawing connections across borders in CSCW and HCI research, design, and practice. To enable contributions in these fields to cross cultures and geographies in productive ways, we draw attention to commonalities in the struggles and processes of resistance operating in different contexts of study. To do this, we present lessons learned from conducting three comparative studies in six contexts, which were located across various borders. The primary contribution of our analysis is to leverage a feminist solidarity-based approach towards extending conversations on comparative, transnational, and feminist CSCW and HCI research, design, and practice. Our focus remains on resource-constrained regions across the world, in both the Global North and South.

Tangibly understanding intangible complexities: Designing for distributed autonomous organizations

Abstract New digital technologies such as Blockchain and smart contracting are rapidly changing the face of value exchange, and present new opportunities and challenges for designers. Designers and data specialists are at the forefront of exploring new ways of exchanging value, using Blockchain, cryptocurrencies, smart contracting and the direct exchanges between things made possible by the Internet of Things (Tallyn et al. 2018; Pschetz et al. 2019). For researchers and designers in areas of Human Computer Interaction (HCI) and Interaction Design to better understand and explore the implications of these emerging and future technologies as Distributed Autonomous Organisations (DAOs) we delivered a workshop at the ACM conference Designing Interactive Systems (DIS) in Edinburgh in 2017 (Nissen et al. 2017). The workshop aimed to use the lens of DAOs to introduce the principle that products and services may soon be owned and managed collectively and not by one person or authority, thus challenging traditional concepts of ownership and power. This workshop builds on established HCI research exploring the role of technology in financial interactions and designing for the rapidly changing world of technology and value exchange (Kaye et al. 2014; Malmborg et al. 2015; Millen et al. 2015; Vines et al. 2014). Beyond this, the HCI community has started to explore these technologies beyond issues of finance, money and collaborative practice, focusing on the implications of these emerging but rapidly ascending distributed systems in more applied contexts (Elsden et al. 2018a). By bringing together designers and researchers with different experiences and knowledge of distributed systems, the aim of this workshop was two-fold. First, to further understand, develop and critique these new forms of distributed power and ownership and second, to practically explore how to design interactive products and services that enable, challenge or disrupt existing and emerging models.

A body sensor data fusion and deep recurrent neural network-based behavior recognition approach for robust healthcare

Recently, human healthcare from body sensor data has been getting remarkable research attentions by a huge range of human-computer interaction and pattern analysis researchers due to its practical applications such as smart health care systems. For example, smart wearable-based behavior recognition system can be used to assist the rehabilitation of patients in a smart clinic to improve the rehabilitation process and to prolong their independent life. Although there are many ways of using distributed sensors to monitor vital signs and behavior of people, physical human action recognition via body sensors provides valuable data regarding an individual's functionality and lifestyle. In this work, we propose a body sensor-based system for behavior recognition using deep Recurrent Neural Network (RNN), a promising deep learning algorithm based on sequential information. We perform data fusion from multiple body sensors such as electrocardiography (ECG), accelerometer, magnetometer, etc. The extracted features are further enhanced via kernel principal component analysis (KPCA). The robust features are then used to train an activity RNN, which is later used for behavior recognition. The system has been compared against the conventional approaches on three publicly available standard datasets. The experimental results show that the proposed approach outperforms the available state-of-the-art methods.

Space Food Experiences: Designing Passenger's Eating Experiences for Future Space Travel Scenarios

Given the increasing possibilities of short- and long-term space travel to the Moon and Mars, it is essential not only to design nutritious foods but also to make eating an enjoyable experience. To date, though, most research on space food design has emphasized the functional and nutritional aspects of food, and there are no systematic studies that focus on the human experience of eating in space. It is known, however, that food has a multi-dimensional and multisensorial role in societies and that sensory, hedonic, and social features of eating and food design should not be underestimated. Here, we present how research in the field of Human-Computer Interaction (HCI) can provide a user-centered design approach to co-create innovative ideas around the future of food and eating in space, balancing functional and experiential factors. Based on our research and inspired by advances in human-food interaction design, we have developed three design concepts that integrate and tackle the functional, sensorial, emotional, social, and environmental/ atmospheric aspects of “eating experiences in space”. We can particularly capitalize on recent technological advances around digital fabrication, 3D food printing technology, and virtual and augmented reality to enable the design and integration of multisensory eating experiences. We also highlight that in future space travel, the target users will diversify. In relation to such future users, we need to consider not only astronauts (current users, paid to do the job) but also paying customers (non-astronauts) who will be able to book a space holiday to the Moon or Mars. To create the right conditions for space travel and satisfy those users, we need to innovate beyond the initial excitement of designing an “eating like an astronaut” experience. To do so we can draw upon prior HCI research in human-food interaction design and build on insights from food science and multisensory research, particularly research that has shown that the environments in which we eat and drink, and their multisensory components, can be crucial for an enjoyable food experience.

When computers take the lead: The automation of leadership

The importance of technology in the workplace has been, and continues to be, on an upward trajectory. Technological progress allows more and more functions once performed by humans to be automated. Theoretical conceptualizations in human-computer interaction (HCI) covered the evolution of computers from ‘tools’ to ‘partners’ in interaction with humans at work. However, nowadays computers have also begun to take over leadership functions, guiding and commanding human workers. We argue that conceptual coverage is in danger of falling short of this development and the implied profound change in hierarchy.To close this gap, we propose the paradigm of ‘computers as leaders’ and call for a scientific discourse of computers performing leadership functions. Building on research in HCI and human-human leadership, we suggest a definition of computer-human leadership and a respective structural model, entangling interaction roles of the different human and computer agents involved. Moreover, we discuss criteria for evaluating automated leadership systems and questions of function allocation, before we bring our propositions together in a theoretical model depicting how humans come to accept and follow a computer leader. Finally, we discuss implications of the proposed paradigm and call for awareness of ethical issues.

Cooperative Sensing and Wearable Computing for Sequential Hand Gesture Recognition

Hand gestures recognition (HGR) has been considered as one of the crucial research fields of human–computer interaction (HCI). Computer vision is a very active research field in the HGR, traditional vision-based methods, which used camera and ultrasonic/optical sensor to collect the videos or images of the hand gestures shown by participants, have some limitations, such as fixed in-lab location, complex lighting conditions, and cluttered backgrounds. In order to provide new approaches, we described the development of a novel hand gesture recognition system that combined wearable armband and smart glove made by customizable pressure sensor arrays to detect sequential hand gestures. A deep learning technique long short-term memory (LSTM) algorithm had been computed to build an effective model to classify hand gestures by training and testing the collected inertial measurement unit (IMU), electromyographic (EMG), and finger and palm’s pressure data. Furthermore, we built a relatively large database of ten sequential hand gestures consisted by five dynamic gestures and five air gestures collected from ten participants. Our experimental results showed an outstanding classification performance of the proposed LSTM algorithm. These findings have promising implications for sequential hand gesture recognition and the HCI research status.

A Review Paper on Student Computer Interaction for Student

DOne of the main aims in human-computer interaction research is to develop an ability to recognize affective state of the user. Such a capability is necessary to have a more human-like nature in humancomputer interaction.