Screen-based Interaction Research Articles

Navigating Borders During the Pandemic: A Collaborative Multi-Sited Approach

In this contribution, we examine a virtual, cross-campus collaboration during the fall of 2020. Our goal was for students to develop deeper knowledge of the materiality of borders through course content as well as getting to know each other and their different contexts in iterative ways. We approached this goal by intentionally seeking to engage our students, who were facing virtual classes for the second or third time given the Covid-19 pandemic. We knew students struggled with online platforms in previous terms and aimed to provide a classroom experience that recognized the limits of screen-based interaction and the multitude of precarities accentuated by the pandemic. While attending our classes, students were also facing the interlocking effects of Covid-19 including illness and death in their families, effects on their employment and mobility, and associated mental health struggles. Thus, community building within and across these classes was key as we navigated teaching online. Finally, we address our social location regarding course content given our complicated histories of migration and identities as undocuscholars: growing up and attending higher education undocumented in the U.S. (though no longer undocumented).

An Approach for Measuring The Effects of Augmented Reality on Human Performance in Maritime Navigation

A lack of Situation Awareness in maritime navigation has been shown to be a reoccurring contributory cause of vessel collisions at sea. Vessel navigation and operational tasks have progressively moved towards screen-based interactions and monitoring, leading to increased navigator Head-Down Time and decreased navigator performance. Augmented Reality enables new and different opportunities to display operational data, which may reduce Head-Down Time and better support navigator Situation Awareness. However, research and development of Augmented Reality solutions in maritime operations are still in the preliminary stages. This paper presents a multidisciplinary approach towards developing and testing Augmented Reality solutions for maritime applications. We describe a methodology for investigating the effects of Augmented Reality on human performance during maritime navigation scenarios and how this data feeds back in an iterative design and development process using a user-centred approach for maritime digitalization and new technology initiatives.

COVIDware: Designing Interactive Everyday Things as Tangible Homeware for Social Isolation

This paper describes our collaborative journey of creating everyday interactive artefacts to help us think, reflect, and live through self-isolation. Through a co-design approach, we designed interactive homeware objects (that we collectively refer to as ‘COVIDware’) to address the challenges of isolation during the pandemic. Five artefacts were developed by self-isolated designers as interactive art installations. We discuss how each creator reflected on her design concept, process, and encounter through concepts of critical making, speculation, and engagement via in-the-isolated-wild deployments. By empowering early researchers/enthusiasts to design ‘with’ smart-materials, and off-the-shelf items, we reflect on how these homey interfaces can enhance people’s wellbeing beyond screen-based interactions. Despite not collaborating in the making process, our findings from the designer’s making process show how all the designed artefacts shared attributes of biophilic design, imperfection, and unconventional interactions with the overarching goal of promoting wellbeing, and meaningful connection with nature, self, and others.

It's Good to Talk: A Comparison of Using Voice Versus Screen-Based Interactions for Agent-Assisted Tasks

Voice assistants have become hugely popular in the home as domestic and entertainment devices. Recently, there has been a move towards developing them for work settings. For example, Alexa for Business and IBM Watson for Business were designed to improve productivity, by assisting with various tasks, such as scheduling meetings and taking minutes. However, this kind of assistance is largely limited to planning and managing user's work. How might they be developed to do more by way of empowering people at work? Our research is concerned with achieving this by developing an agent with the role of a facilitator that assists users during an ongoing task. Specifically, we were interested in whether the modality in which the agent interacts with users makes a difference: How does a voice versus screen-based agent interaction affect user behavior? We hypothesized that voice would be more immediate and emotive, resulting in more fluid conversations and interactions. Here, we describe a user study that compared the benefits of using voice versus screen-based interactions when interacting with a system incorporating an agent, involving pairs of participants doing an exploratory data analysis task that required them to make sense of a series of data visualizations. The findings from the study show marked differences between the two conditions, with voice resulting in more turn-taking in discussions, questions asked, more interactions with the system and a tendency towards more immediate, faster-paced discussions following agent prompts. We discuss the possible reasons for why talking and being prompted by a voice assistant may be preferable and more effective at mediating human-human conversations and we translate some of the key insights of this research into design implications.

Natural notification system for the interior of shared car

The paper describes the design process, hardware prototyping and user testing of an innovative user interface notification system for the interior of a shared autonomous car. We describe the idea and need for this design, its hardware and software functions, and prototype production. The key innovation is in exploring possibilities of applying the so-called natural user interface approach - meaning avoiding screen-based interactions and graphical user interfaces in favor of gesture, voice or glance interaction.

Multimodal Observation and Classification of People Engaged in Problem Solving: Application to Chess Players

In this paper we present the first results of a pilot experiment in the interpretation of multimodal observations of human experts engaged in solving challenging chess problems. Our goal is to investigate the extent to which observations of eye-gaze, posture, emotion and other physiological signals can be used to model the cognitive state of subjects, and to explore the integration of multiple sensor modalities to improve the reliability of detection of human displays of awareness and emotion. Domains of application for such cognitive model based systems are, for instance, healthy autonomous ageing or automated training systems. Abilities to observe cognitive abilities and emotional reactions can allow artificial systems to provide appropriate assistance in such contexts. We observed chess players engaged in problems of increasing difficulty while recording their behavior. Such recordings can be used to estimate a participant’s awareness of the current situation and to predict ability to respond effectively to challenging situations. Feature selection has been performed to construct a multimodal classifier relying on the most relevant features from each modality. Initial results indicate that eye-gaze, body posture and emotion are good features to capture such awareness. This experiment also validates the use of our equipment as a general and reproducible tool for the study of participants engaged in screen-based interaction and/or problem solving.

The Body of the Machine: Computer Interfaces in American Popular Cinema since 1982

Human-computer interaction in the late twentieth and early twentyfirst century has been aided by the graphical user interface (GUI). Cinematic representations of the GUI have reflected its pervasive quotidian presence and expanded beyond to create spectacular technophilic Hollywood blockbusters. The screen-based interaction, which computers and the cinema share, offers an important point of reflection upon the embodied relationship among screen, spectator, and the onscreen computer user. By examining cinematic representations of hackers, experts, and users and the extent to which they manipulate or submit to the computer’s vision, this article considers the possible future interaction between spectator and the digital medium of cinema.

Head Up Games: combining the best of both worlds by merging traditional and digital play

Current pervasive games are mostly location-aware applications, played on handheld computing devices. Considering pervasive games for children, it is argued that the interaction paradigm existing games support limits essential aspects of outdoor play like spontaneous social interaction, physical movement, and rich face-to-face communication. We present a new genre of pervasive games conceived to address this problem, that we call Head Up Games (HUGs) to underline that they liberate players from facing down to attend to screen-based interactions. The article discusses characteristics of HUG and relates them to existing genres of pervasive games. We present lessons learned during the design and evaluation of three HUG and chart future challenges.