Interaction Modalities Research Articles

PerDis 2017

Learn about the tutorials, keynote, and papers presented at the 6th ACM International Symposium on Pervasive Displays (PerDis 2017), which took place in Lugano, Switzerland. Topics ranged from introducing public display applications, platforms, and frameworks to proposing interaction modalities and understanding the audience behavior, engagement, cognition, and performance.

When Smart Devices Interact With Pervasive Screens

The meeting of pervasive screens and smart devices has witnessed the birth of screen-smart device interaction (SSI), a key enabler to many novel interactive use cases. Most current surveys focus on direct human-screen interaction, and to the best of our knowledge, none have studied state-of-the-art SSI. This survey identifies three core elements of SSI and delivers a timely discussion on SSI oriented around the screen, the smart device, and the interaction modality. Two evaluation metrics (i.e., interaction latency and accuracy) have been adopted and refined to match the evaluation criterion of SSI. The bottlenecks that hinder the further advancement of the current SSI in connection with this metrics are studied. Last, future research challenges and opportunities are highlighted in the hope of inspiring continuous research efforts to realize the next generation of SSI.

Improving the Accessibility of Mobile OCR Apps Via Interactive Modalities.

Mobile optical character recognition (OCR) apps have come of age. Many blind individuals use them on a daily basis. The usability of such tools, however, is limited by the requirement that a good picture of the text to be read must be taken, something that is difficult to do without sight. Some mobile OCR apps already implement auto-shot and guidance mechanisms to facilitate this task. In this paper, we describe two experiments with blind participants, who tested these two interactive mechanisms on a customized iPhone implementation. These experiments bring to light a number of interesting aspects of accessing a printed document without sight, and enable a comparative analysis of the available interaction modalities.

DataTweener

With the development and advancement of new data interaction modalities, data exploration and analysis has become a highly interactive process situating the user in a session of successive queries. With rapidly changing results, it becomes difficult for the end user to fully comprehend transformations, especially the transforms corresponding to complex queries. We introduce "data tweening" as an informative way of visualizing structural data transforms, presenting the users with a series of incremental visual representations of a resultset transformation. We present transformations as ordered sequences of basic structural transforms and visual cues. The sequences are generated using an automated framework which utilizes differences between the consecutive resultsets and queries in a query session. We evaluate the effectiveness of tweening as a visualization method through a user study.

Robust classification of biological samples in atomic force microscopy images via multiple filtering cooperation

The morphological changes induced by engineered nanomaterials (ENMs) in cellular samples are a key clue to evaluate the impact of these materials on human health. The investigation of the complexity of the interaction among nanoparticles and cellular molecules requires cutting-edge instrumentation and dedicated procedures. To this regard, atomic force microscopy (AFM) is a leading imaging technique that has the peculiarities of high resolution and direct relationship with 3D cellular morphology. Expert human operators, however, are still required to manage most of the AFM-based analysis, thus introducing subjective bias and allowing a limited number of trials. The modality of interaction and the resulting topographic differences are often not a priori known. Also, the presence of artefacts in AFM images may affect the derived conclusions. In this work, we propose a robust and flexible strategy to analyse AFM topography images with single click-select actions by the human operator. The proposed system allows for not only morphological studies and quantification of the changes occurring in cellular samples in the presence of nanomaterials, but also for the investigation of diversified experiments in more flexible application domains. As a proof of concepts, samples of human EA. hy926 endothelial cells exposed to carbon nanotubes are used to demonstrate the effectiveness of the proposed solution. The system is also tested against various AFM-artefact and noise scenarios and the robustness of its discrimination capability is verified.

Explorations of a Very-large-screen Digital Library Interface

While digital libraries accommodate remote access via the web and mobile devices, their physical presence tends to be minuscule. A locally-developed prototype digital library application called DLib Wall, connected to MultiTaction display hardware from MultiTouch Ltd., is an attempt to create a rich and engaging onsite presence for the digital collections of the University of Nevada, Las Vegas Libraries. DLib Wall is one of the first applications of its kind, and demonstrates several relatively unexplored interaction modalities. In late 2014, it was deployed in the Goldfield Room — a conference room in the Lied Library — on a wall-mounted array of six 42-inch touch displays. DLib Wall represents both application development and the work of a collaborative team charged with creating a large, interactive, and content-rich media wall experience on a limited resource budget. The group's work included: evaluating and recommending technologies, managing the custom development for the DLib Wall application, and creating an initial content plan for the public rollout. This article highlights the technical aspects of the application while sharing key decision points that the team encountered in the process of bringing the project from conception to completion.

Video-based arm motion estimation and interaction with fuzzy predictive control

Multi-projector display systems are gaining popularity for use in immersive virtual reality applications and scientific visualization. While recent work has addressed the issues of human interfaces to hide the distributed nature of those systems, there has been relatively little work on natural interactive modalities. In this paper, based on the discrete characteristics of nodes distribution and the spatio-temporal coherence of the user’s movement, we propose a non-contact interaction solution for multi-projector display system. Utilizing a virtual three-dimensional interactive rectangular parallelepiped, we establish correspondence between the virtual scene and the user’s arm position information. For robustly tracking the user’s arm position, an arm motion estimation method is designed based on the fuzzy predictive control theory. To verify the efficiency and accuracy of the proposed method, various motion estimation algorithms were tested with and without the fuzzy predictive control theory to stabilize the output.

Polymodal Menus

This paper presents a model-based approach for designing Polymodal Menus, a new type of multimodal adaptive menu for small screen graphical user interfaces where item selection and adaptivity are responsive to more than one interaction modality: a menu item can be selected graphically, tactilely, vocally, gesturally, or any combination of them. The prediction window containing the most predicted menu items by assignment, equivalence, or redundancy is made equally adaptive. For this purpose, an adaptive menu model maintains the most predictable menu items according to various prediction methods. This model is exploited throughout various steps defined on a new Adaptivity Design Space based on a Perception-Decision-Action cycle com-ing from cognitive psychology. A user experiment compares four conditions of Polymodal Menus (graphical, vocal, gestural, and mixed) in terms of menu selection time, error rate, user subjective satisfaction and user preference, when item prediction has a low or high level of accuracy. Polymodal Menus offer alternative input/output modalities to select menu items in various contexts of use, especially when graphical modality is constrained.

Intensifying Emotional Reactions via Tactile Gestures in Immersive Films

The film industry continuously strives to make visitors’ movie experience more immersive and thus, more captivating. This is realized through larger screens, sophisticated speaker systems, and high quality 2D and 3D content. Moreover, a recent trend in the film industry is to incorporate multiple interaction modalities, such as 4D film, to simulate rain, wind, vibration, and heat, in order to intensify viewers’ emotional reactions. In this context, humans’ sense of touch possesses significant potential for intensifying emotional reactions for the film experience beyond audio-visual sensory modalities. This article presents a framework for authoring tactile cues (tactile gestures as used in this article) and enabling automatic rendering of said gestures to intensify emotional reactions in an immersive film experience. To validate the proposed framework, we conducted an experimental study where tactile gestures are designed and evaluated for the ability to intensify four emotional reactions: high valence-high arousal, high valence-low arousal, low valence-high arousal, and low valence-low arousal. Using a haptic jacket, participants felt tactile gestures that are synchronized with the audio-visual contents of a film. Results demonstrated that (1) any tactile feedback generated a positive user experience; (2) the tactile feedback intensifies emotional reactions when the audio-visual stimuli elicit clear emotional responses, except for low arousal emotional response since tactile gestures seem to always generate excitement; (3) purposed tactile gestures do not seem to significantly outperform randomized tactile gesture for intensifying specific emotional reactions; and (4) using a haptic jacket is not distracting for the users.

An Exploration of the Benefits of an Animallike Robot Companion with More Advanced Touch Interaction Capabilities for Dementia Care

Animallike robot companions such as robotic seal Paro are increasingly used in dementia care due to the positive effects that interaction with these robots can have on the well-being of these patients. Touch is one of the most important interaction modalities for patients with dementia and can be a natural way to interact with animallike robots. To advance the development of animallike robots we explored in what ways people with dementia could benefit from interaction with an animallike robot with more advanced touch recognition capabilities and which touch gestures would be important in their interaction with Paro. In addition we explored which other target groups might benefit from interaction with animallike robots with more advanced interaction capabilties. In this study we administered a questionnaire and conducted interviews with two groups of health care providers who all worked in a geriatric psychiatry department. One group used Paro in their work (i.e., the expert group; $n=5$) while the other group had no experience with the use of animallike robot (i.e., the layman group; $n=4$). The results showed that health care providers perceived Paro as an effective intervention to improve the well-being of people with dementia. Examples of usages for Paro that were mentioned were providing distraction, interrupting problematic behaviors and stimulating communication. Furthermore, the care providers indicated that people with dementia (would) use mostly positive forms of touch and speech to interact with Paro. Paro's auditory responses were criticized because they can overstimulate the patients. Additionally, the care providers argued that social interactions with Paro are currently limited and therefore the robot does not meet the needs of a broader audience such as healthy elderly people that still live in their own homes. The development of robot pets with more advanced social capabilities such as touch and speech recognition might result in more intelligent interactions which could help to better adapt to the needs of people with dementia and could make interactions more interesting for a broader audience. Moreover, the robot's response modalities and its appearance should match the needs of to the target group.

GraSp: Combining Spatially‐aware Mobile Devices and a Display Wall for Graph Visualization and Interaction

AbstractGoing beyond established desktop interfaces, researchers have begun re‐thinking visualization approaches to make use of alternative display environments and more natural interaction modalities. In this paper, we investigate how spatially‐aware mobile displays and a large display wall can be coupled to support graph visualization and interaction. For that purpose, we distribute typical visualization views of classic node‐link and matrix representations between displays. The focus of our work lies in novel interaction techniques that enable users to work with personal mobile devices in combination with the wall. We devised and implemented a comprehensive interaction repertoire that supports basic and advanced graph exploration and manipulation tasks, including selection, details‐on‐demand, focus transitions, interactive lenses, and data editing. A qualitative study has been conducted to identify strengths and weaknesses of our techniques. Feedback showed that combining mobile devices and a wall‐sized display is useful for diverse graph‐related tasks. We also gained valuable insights regarding the distribution of visualization views and interactive tools among the combined displays.

Design and implementation of a novel architecture for physical human-UAV interaction

Interaction between humans and unmanned aerial vehicles is a promising field for future applications. However, current interfacing paradigms either imply the presence of intermediary hardware as monitors, joysticks and haptic devices, or are limited to visual/auditory channels with hand gestures, voice recognition, or interpretation of face poses and body postures. Another paradigm, physical human–robot interaction, which is based on mutual exchange of forces, is popular when dealing with robotic arms and humanoids, while unmanned aerial vehicles are usually considered too dangerous and lack proper interaction surfaces to exchange forces. In this paper, we address the problem of physical human–unmanned aerial vehicle interaction and we propose a straightforward approach to allow a human to intuitively command an unmanned aerial vehicle through exchanges of forces. Using a residual based estimator, we estimate the external forces and torques acting on the unmanned aerial vehicle. Through the employment of a sensor ring, we are able to separate the human interaction forces from additional disturbances as wind and parameter uncertainties. This knowledge is used inside a control framework where the human is allowed to change the desired trajectory by simply applying forces on the unmanned aerial vehicle. The system is validated with multiple hardware-in-the-loop simulations and experiments in which we try different interaction modalities.

SeleCon: Scalable IoT Device Selection and Control Using Hand Gestures.

Although different interaction modalities have been proposed in the field of human-computer interface (HCI), only a few of these techniques could reach the end users because of scalability and usability issues. Given the popularity and the growing number of IoT devices, selecting one out of many devices becomes a hurdle in a typical smarthome environment. Therefore, an easy-to-learn, scalable, and non-intrusive interaction modality has to be explored. In this paper, we propose a pointing approach to interact with devices, as pointing is arguably a natural way for device selection. We introduce SeleCon for device selection and control which uses an ultra-wideband (UWB) equipped smartwatch. To interact with a device in our system, people can point to the device to select it then draw a hand gesture in the air to specify a control action. To this end, SeleCon employs inertial sensors for pointing gesture detection and a UWB transceiver for identifying the selected device from ranging measurements. Furthermore, SeleCon supports an alphabet of gestures that can be used for controlling the selected devices. We performed our experiment in a 9m-by-10m lab space with eight deployed devices. The results demonstrate that SeleCon can achieve 84.5% accuracy for device selection and 97% accuracy for hand gesture recognition. We also show that SeleCon is power efficient to sustain daily use by turning off the UWB transceiver, when a user's wrist is stationary.

Combinatorial Principle in the Use of Drum (Kendhang) Formulae in Gamelan Music of Yogyakarta. From Prescriptive Models to the Interpretation of a Composition: the Case of Kendhang Pinatut.

In this paper, I address the question to the use of drums (kendhang) in the traditional Gamelan music of Yogyakarta, by presenting some prescriptive models (or formulas). I illustrate how, the use of different prescriptive models in a composition follow what I labeled as “Combinatorial Principle”. In order to describe the essential elements of this principle, I will analyze the modalities of interaction between a very flexible drum formula (known as pinatut) and three other prescriptive models for drum within some exemplary pieces of traditional Gamelan music. The concept of combinatorial principle illustrated in these pages, on the one hand explains the way of interaction between the drum’s rhythmic formulas and their capacity to influence the choices made by the entire orchestra during a performance; on the other hand, through this principle we are able to trace a path that attempts to understand the “deep structures” that are the basics of making music in Gamelan tout court. Through the perspective of the combinatorial principle it is possible to analyze the prescriptive models and techniques of many other instruments of the Gamelan of central Java.

A mobile platform for controlling and interacting with a do-it-yourself smart eyewear

PurposeSmart eyewear, such as augmented or virtual reality headset, allows the projection of virtual content through a display worn on the user’s head. This paper aims to present a mobile platform, named “CARTON”, which transforms a smartphone into smart eyewear, following a do-it-yourself (DIY) approach. This platform is composed of three main components: a blueprint to build the hardware prototype with very simple materials and regular tools; a software development kit (SDK) to help with the development of new applications (e.g. augmented reality app); and, finally, a second SDK (ControlWear) to interact with mobile applications through a Smartwatch.Design/methodology/approachUser experiments were conducted, in which participants were asked to create, by themselves, the CARTON’s hardware part and perform usability tests with their own creation. A second round of experimentation was conducted to evaluate three different interaction modalities.FindingsQualitative user feedback and quantitative results prove that CARTON is functional and feasible to anyone, without specific skills. The results also showed that ControlWear had the most positive results, compared with the other interaction modalities, and that user interaction preference would vary depending on the task.Originality/valueThe authors describe a novel way to create a smart eyewear available for a wide audience around the world. By providing everything open-source and open-hardware, they intend to solve the reachability of technologies related to smart eyewear and aim to accelerate research around it.

Unsupervised Gesture Segmentation by Motion Detection of a Real-Time Data Stream

Continuous and real-time gesture spotting is a key factor in the development of novel human–machine interaction modalities. Gesture recognition can be greatly improved with previous reliable segmentation. This paper introduces a new unsupervised threshold-based hand/arm gesture segmentation method to accurately divide continuous data streams into dynamic and static segments from unsegmented and unbounded input data. This segmentation may reduce the number of wrongly classified gestures in real-world conditions. The proposed approach identifies sudden inversions of movement direction, which are a cause of oversegmentation (excessive segmentation). This is achieved by the analysis of velocities and accelerations numerically derived from positional data. A genetic algorithm is used to compute feasible thresholds from calibration data. Experimental tests with three different subjects demonstrated an average oversegmentation error of 2.70% in a benchmark for motion segmentation with a feasible sliding window size.

Piloting a mobile health intervention to increase physical activity for adolescents with ADHD.

Physical activity (PA) reduces symptoms of Attention Deficit Hyperactivity Disorder (ADHD); interventions to increase PA may improve functioning and health for adolescents with ADHD. Mobile health (mHealth) technology and social media constitute promising interactive modalities for engaging adolescents—who are at highest risk for ADHD treatment drop-out—in interventions to increase PA. The current pilot study evaluated feasibility and acceptability of an innovative intervention incorporating an mHealth-linked wearable activity tracker (Fitbit Flex) and a Facebook group to increase PA among adolescents with ADHD. 11 adolescents diagnosed with ADHD (age 14–18, m = 15.5; 54% female) participated in a 4-week trial utilizing the Fitbit Flex in conjunction with (1) weekly personalized step count goals (2) social support through a Facebook group and (3) daily text messages about PA. The study took place in the greater Seattle, Washington area in the fall of 2015. Adolescents completed online surveys twice per week to rate their ADHD symptoms and positive and negative mood states, and parents rated adolescent ADHD symptoms weekly.Participants were adherent to the study protocol and acceptability of the intervention was high. Linear mixed models indicated that participants significantly increased their average weekly steps over the course of the study and demonstrated improvements in both adolescent and parent-reported ADHD Inattentive symptoms.Results indicate that this mHealth intervention is engaging and promising for increasing PA among adolescents with ADHD, and warrant further study. Implications for improving ADHD symptoms and overall functioning for this undertreated population are discussed.

Implementation Plan in Response to the Executive Board-Endorsed Recommendations for the IEO Evaluation Report—The IMF and the Crises in Greece, Ireland, and Portugal

This paper sets out Management’s response to the Independent Evaluation Office’s (IEO) report The IMF and the Crises in Greece, Ireland, and Portugal as part of the Management and staff follow-up to an IEO evaluation. The implementation plan proposes specific actions to address the recommendations endorsed by the Executive Board in its July 19, 2016 discussion. The implementation plan notes the Board’s endorsement of the principles that the IMF’s technical analysis should remain independent, that existing policies should be followed and should not be changed without careful deliberation, and that the Executive Board and Management should reaffirm their commitment to accountability and transparency, as well as the role of independent evaluation in fostering good governance. As to the IEO’s specific recommendations supported by the Board, the implementation plan proposes actions to address the need for the Fund to clarify how guidelines on program design apply to currency union members, the need for the Fund to establish a policy on cooperation with regional financing arrangements, and the need to further strengthen Fund cooperation with the IEO, including on modalities of interactions between the IEO and Fund staff and the IEO’s access to information. The implementation of some of the proposed actions is already underway and some actions have been included in the Fall 2016 Work Program. The paper also explains how implementation will be monitored by the Executive Board.

Teratogenic hazard of BPEI-coated silver nanoparticles to Xenopus laevis

Silver nanoparticles (AgNPs) are among the most exploited antimicrobial agents and are used in many consumer products. Size and surface reactivity are critical physico-chemical properties responsible for NPs toxicity, and surface coatings, often used to functionalize or stabilize AgNPs, can influence their toxic profile and biocompatibility. In the current study the developmental toxicity of (1) negatively charged citrate-coated AgNPs (Cit-AgNPs), (2) positively charged branched polyethylenimine-coated AgNPs (BPEI-AgNPs), and (3) Ag+ (from 0.0625 to 0.75 mg Ag/L) was investigated by the standard Frog Embryo Teratogenesis Assay – Xenopus (FETAX). In order to identify the most sensitive developmental phase, embryos were also exposed during different embryonic stages. Morphological and bio-physical studies were performed to characterize tissue lesions and NP uptake. The results suggest that Ag+ was strongly embryo-lethal. Contrary to Cit-AgNPs, the positively charged BPEI-AgNPs exert a concentration-dependent effect on lethality and malformations of embryos. The BPEI-AgNPs showed the highest teratogenic index (TI = 1.6), pointing out the role of functional coating in determining the developmental hazard. The highest susceptibility to BPEI-AgNPs was during early embryogenesis, when embryos are still enclosed in the fertilization envelope, and the post-stomodeum opening stages, when NPs ingestion occurs. In BPEI-AgNPs treated larvae, the histological examination revealed irregular intestinal diverticula coupled with edematous connective tissue. Small NPs aggregates are mapped throughout the intestinal mucosa and secondary target organs by two-photon excitation microscopy. We conclude that a teratogenic risk may be associated with BPEI-AgNPs exposure, but the modality of NP-tissue interactions and the teratogenic mechanism need further investigations to be better defined.

The benefits of distance and mediation: How people react to conflicts in video chat vs. FtF

Conflicts come as natural elements in human relationships and they are experienced in face-to-face as well as mediated interactions. Yet, it is unclear in which way and to which degree the modality of interaction influences the experience, the process, and the outcome of conflict communication. Comparing face-to-face and video chat encounters, the current experiment examined whether different communication modalities (video chat: mediated-distal vs. face-to-face: non-mediated-proximal) affect emotional arousal, partner evaluation, and communication satisfaction in conflict-laden interactions. Arousal was measured via self-report as well as physiological measures (interbeat interval, galvanic skin response). Results show that people who interacted via video chat reported less arousal than those who interacted face-to-face although there was no significant difference in physiological arousal measures. Also, those who interacted via video chat rated their partner and their conversation more positively. The findings suggest that, although some nonverbal signals were conveyed in both settings, the lack of mere physical co-presence in mediated encounters might be beneficial for conflict resolution.