Tangible Interface Research Articles

AdapTUI: Adaptation of Geometric-Feature-Based Tangible User Interfaces in Augmented Reality

With the advents in geometry perception and Augmented Reality (AR), end-users can customize Tangible User Interfaces (TUIs) that control digital assets using intuitive and comfortable interactions with physical geometries (e.g., edges and surfaces). However, it remains challenging to adapt such TUIs in varied physical environments while maintaining the same spatial and ergonomic affordance. We propose AdapTUI, an end-to- end system that enables an end-user to author geometric-based TUIs and automatically adapts the TUIs when the user moves to a new environment. Leveraging a geometry detection module and the spatial awareness of AR, AdapTUI first lets users create custom mappings between geometric features and digital functions. Then, AdapTUI uses an optimization-based adaptation framework, which considers both the geometric variations and human-factor nuances, to dynamically adjust the attachment of the user-authored TUIs. We demonstrate three application scenarios where end-users can utilize TUIs at different locations, including portable car play, efficient AR workstation, and entertainment. We evaluated the effectiveness of the adaptation method as well as the overall usability through a comparison user study (N=12). The satisfactory adaptation of the user-authored TUIs and the positive qualitative feedback demonstrate the effectiveness of our system.

MoiréTag: A Low-Cost Tag for High-Precision Tangible Interactions without Active Components

In this paper, we present MoiréTag—a novel tag-like device that magnifies displacement without active components for indirect sensing of subtle tangible interactions. The device consists of two overlapping layers of stripe patterns with distinct pattern frequencies. These layers create Moiré fringes that can move faster than the actual movement of a layer. Using a customized image processing pipeline, we show that MoiréTag can reliably detect sub-mm movement in real-time (mean error = 0.043 mm) under varying lighting conditions, camera angles, and camera distances. We also demonstrate five applications of MoiréTag to showcase its potential as a low-cost solution to capture and monitor small changes in movement and other physical properties, such as force and volume, by converting them into displacement.

Evaluation of Haptic Textures for Tangible Interfaces for the Tactile Internet

Every texture in the real world provides us with the essential information to identify the physical characteristics of real objects. In addition to sight, humans use the sense of touch to explore their environment. Through haptic interaction we obtain unique and distinct information about the texture and the shape of objects. In this paper, we enhance X3D 3D graphics files with haptic features to create 3D objects with haptic feedback. We propose haptic attributes such as static and dynamic friction, stiffness, and maximum altitude that provide the optimal user experience in a virtual haptic environment. After numerous optimization attempts on the haptic textures, we propose various haptic geometrical textures for creating a virtual 3D haptic environment for the tactile Internet. These tangible geometrical textures can be attached to any geometric shape, enhancing the haptic sense. We conducted a study of user interaction with a virtual environment consisting of 3D objects enhanced with haptic textures to evaluate performance and user experience. The goal is to evaluate the realism and recognition accuracy of each generated texture. The findings of the study aid visually impaired individuals to better understand their physical environment, using haptic devices in conjunction with the enhanced haptic textures.

Evaluating Force-based Haptics for Immersive Tangible Interactions with Surface Visualizations.

Haptic feedback provides an essential sensory stimulus crucial for interaction and analyzing three-dimensional spatio-temporal phenomena on surface visualizations. Given its ability to provide enhanced spatial perception and scene maneuverability, virtual reality (VR) catalyzes haptic interactions on surface visualizations. Various interaction modes, encompassing both mid-air and on-surface interactions-with or without the application of assisting force stimuli-have been explored using haptic force feedback devices. In this paper, we evaluate the use of on-surface and assisted on-surface haptic modes of interaction compared to a no-haptic interaction mode. A force-based haptic stylus is used for all three modalities; the on-surface mode uses collision based forces, whereas the assisted on-surface mode is accompanied by an additional snapping force. We conducted a within-subjects user study involving fundamental interaction tasks performed on surface visualizations. Keeping a consistent visual design across all three modes, our study incorporates tasks that require the localization of the highest, lowest, and random points on surfaces; and tasks that focus on brushing curves on surfaces with varying complexity and occlusion levels. Our findings show that participants took almost the same time to brush curves using all the interaction modes. They could draw smoother curves using the on-surface interaction modes compared to the no-haptic mode. However, the assisted on-surface mode provided better accuracy than the on-surface mode. The on-surface mode was slower in point localization, but the accuracy depended on the visual cues and occlusions associated with the tasks. Finally, we discuss participant feedback on using haptic force feedback as a tangible input modality and share takeaways to aid the design of haptics-based tangible interactions for surface visualizations.

Reflections on using the story completion method in designing tangible user interfaces

There are many design techniques to support the co-design of tangible technologies. However, few of these design methods allow the involvement of users at scale and across diverse geographic locations. While popular in psychology, the story completion method (SCM) has only recently started to be adopted within the HCI community. We explore whether SCM can generate meaningful design insights from large, diverse study populations for the design of Tangible User Interfaces (TUIs). Based on the results of two questionnaire studies using SCM, we conclude that the method can be used to generate meaningful design insights. Drawing on a systematic review of 870 TUI papers, we then contextualise the strengths and weaknesses of SCM against commonly used design methods, before reflecting on our experience of using the method across two distinct domains. We discuss the advantages of the method (particularly in terms of the scale and diversity of participation) and the challenges (particularly around constructing meaningful story stems, and developing the correct level of scaffolding to support creativity). We conclude that SCM is particularly suitable to be used in the early stages of the design process to understand the socio-cultural context of deployment.

Effectiveness of mixed reality-based rehabilitation on hands and fingers by individual finger-movement tracking in patients with stroke

BackgroundMixed reality (MR) is helpful in hand training for patients with stroke, allowing them to fully submerge in a virtual space while interacting with real objects. The recognition of individual finger movements is required for MR rehabilitation. This study aimed to assess the effectiveness of updated MR-board 2, adding finger training for patients with stroke.MethodsTwenty-one participants with hemiplegic stroke (10 with left hemiplegia and 11 with right hemiplegia; nine female patients; 56.7 ± 14.2 years of age; and onset of stroke 32.7 ± 34.8 months) participated in this study. MR-board 2 comprised a board plate, a depth camera, plastic-shaped objects, a monitor, a palm-worn camera, and seven gamified training programs. All participants performed 20 self-training sessions involving 30-min training using MR-board 2. The outcome measurements for upper extremity function were the Fugl–Meyer assessment (FMA) upper extremity score, repeated number of finger flexion and extension (Repeat-FE), the thumb opposition test (TOT), Box and Block Test score (BBT), Wolf Motor Function Test score (WMFT), and Stroke Impact Scale (SIS). One-way repeated measures analysis of variance and the post hoc test were applied for the measurements. MR-board 2 recorded the fingers’ active range of motion (AROM) and Dunnett’s test was used for pairwise comparisons.ResultsExcept for the FMA-proximal score (p = 0.617) and TOT (p = 0.005), other FMA scores, BBT score, Repeat-FE, WMFT score, and SIS stroke recovery improved significantly (p < 0.001) during MR-board 2 training and were maintained until follow-up. All AROM values of the finger joints changed significantly during training (p < 0.001).ConclusionsMR-board 2 self-training, which includes natural interactions between humans and computers using a tangible user interface and real-time tracking of the fingers, improved upper limb function across impairment, activity, and participation. MR-board 2 could be used as a self-training tool for patients with stroke, improving their quality of life.Trial registration number: This study was registered with the Clinical Research Information Service (CRIS: KCT0004167).

Trends in the Use of Augmented Reality from Tangible Interfaces in School Learning : A Systematic Literature Review

This study aims to analyze trends in the use of augmented reality from tangible interfaces in the context of formal and informal education. This study employed a systematic literature review (SLR) method adapted from the PRISMA model. This research used the Publish or Perish application to search for various relevant scientific articles taken from international journals, Scopus, with a range of 2019-2023. The data analysis technique for this research used thematic analysis. The development of AR media showed significant improvement, especially in the aspect of increasingly realistic and diverse user interfaces, enabling closer integration with the curriculum and presenting more engaging learning experiences. The interactive and collaborative features of AR applications showed great potential for enhancing student interaction with learning materials through immersive and interactive visual experiences. In addition, such collaborative features could also facilitate cooperation between students, creating a more dynamic and effective learning environment in schools. This research emphasized the importance of using appropriate research methods to deepen the understanding of the development of AR media in education and the potential of AR applications in creating more interactive and collaborative learning experiences in school environments.

Tangible Interaction Design in Music Intervention for Chinese Children with Developmental Dyslexia

With the development of children’s education, parents and teachers pay more and more attention to the learning and training of children with developmental dyslexia (DD). There are many reasons for DD and various intervention tools with a large number of learning costs. In this paper, we aim to analyze the relationship between music, pinyin, and reading, and confirm the theoretical link of "music - Chinese pinyin - reading ability". Then, we sort out a method to design the music intervention tool in this article, which is a new tangible interactive tool for children with DD. This tool is composed of hardware and software and has three modules: music intervention, pinyin intervention, and reading intervention, covering a total of 11 levels and is suitable for classroom teaching, home self-study, and other scenarios, to improve the Chinese pinyin and reading abilities. Subsequently, we conducted a pre-experiment and formal experiment on 20 children with DD to demonstrate the effectiveness of this tool, while using the SUS scale to analyze the evaluation of these children. Finally, we figured that the intervention tool designed in this study has a certain helpful effect on improving the phonetic and reading abilities of children with DD, and is more effective than the three intervention methods used alone. This study has certain reference significance for the designers of intervention tools and also provides new intervention methods for teachers in training institutions. It also provides another possibility for the self-treatment and training of children with developmental dyslexia.

96‐4: Invited Paper: Surface Dial: Enabling Tangible Dual‐handed Interactions on Capacitive Touchscreens

Surface Dial is a mass production PC accessory implementing a tangible interface for on‐and off‐screen interactions. It enables a dual‐handed interaction model in content creation tasks like drawing, video editing, presentation and data visualization. We delve into design considerations, including development of a capacitive approach for on‐screen detection and tracking.

Matti: Tangible User Interface for Engaging Patients in Physical Therapy Towards a Motivating Rehabilitation

Matti: Tangible User Interface for Engaging Patients in Physical Therapy Towards a Motivating Rehabilitation

Happy Click!: Investigating the Use of a Tangible Interface to Facilitate the Three Good Things Positive Psychology Intervention

Abstract The ‘Three Good Things’ (TGT), a structured journaling exercise, has shown great effect in helping improve mental well-being. However, the exercise could be quite tedious and difficult to carry out daily. To improve the user experience of the exercise, we developed ‘Happy Click’, a mobile application which makes use of a physically present smart button to enable users to record their daily TGT in a more engaging manner. Two preliminary exploratory studies were carried out to examine the feasibility of our approach where the effects of the TGT intervention designed around the mobile and tangible happy click concepts were examined. The findings suggested that participants had significantly less stress after carrying out the TGT using the tangible interface. Participants also reported better user experience and acceptance when carrying out the exercise using the tangible interface compared to the mobile and higher levels of attractiveness and stimulation with the mobile TGT compared to the traditional approach. While these promising results should be interpreted with caution due to the limited sample size, they also underscore the potential advantages of our ‘Happy click’ approach and highlight the need for further, larger-scale evaluations to be conducted to validate its efficacy and scalability.

A Narrative Review of the Sociotechnical Landscape and Potential of Computer-Assisted Dynamic Assessment for Children with Communication Support Needs

This paper presents a narrative review of the current practices in assessing learners’ cognitive abilities and the limitations of traditional intelligence tests in capturing a comprehensive understanding of a child’s learning potential. Referencing prior research, it explores the concept of dynamic assessment (DA) as a promising yet underutilised alternative that focuses on a child’s responsiveness to learning opportunities. The paper highlights the potential of novel technologies, in particular tangible user interfaces (TUIs), in integrating computational science with DA to improve the access and accuracy of assessment results, especially for children with communication support needs (CSN), as a catalyst for abetting critical communicative competencies. However, existing research in this area has mainly focused on the automated mediation of DA, neglecting the human element that is crucial for effective solutions in special education. A framework is proposed to address these issues, combining pedagogical and sociocultural elements alongside adaptive information technology solutions in an assessment system informed by user-centred design principles to fully support teachers/facilitators and learners with CSN within the special education ecosystem.

Tangible and Mid-Air Interactions in Hand-Held Augmented Reality for Upper Limb Rehabilitation: An Evaluation of User Experience and Motor Performance

Hand-held augmented reality (AR) offers accessible, interactive rehabilitation options for patients with upper limb motor deficits. Incorporating hand-involved interactions (e.g., tangible and mid-air interactions) into hand-held AR provides patients with intuitive manners to perform rehabilitation exercises mimicking real-world activities. Previous work has shown the importance of user experience and motor performance in rehabilitation systems, but little was known in the literature regarding the impact of hand-involved interactions in hand-held AR on user experience and motor performance in rehabilitation exercises. Hence, this study aims to evaluate user experience and motor performance when using three types of hand-involved interactions in hand-held AR rehabilitation: (1) tangible cube (i.e., a space-multiplexed tangible interaction with a physical cube acting as a real proxy to manipulate a virtual object in the same form); (2) tangible controller (i.e., a time-multiplexed tangible interaction with a physical controller applied to manipulate a virtual object); and (3) hand motion (i.e., a form of mid-air interaction to move a virtual object with hands). Based on the findings from self-report, electroencephalography (EEG), and performance measures, this study reveals the advantages of the tangible cube over the tangible controller, both superior to the hand motion in hand-held AR rehabilitation regarding user experience and motor performance. This study offers new understanding of the advantages and disadvantages of various interaction techniques in hand-held AR rehabilitation, emphasizing crucial design considerations for these systems, with a focus on user experience and motor performance in upper limb rehabilitation.

Engaging Teens Through Playful Tangible Interaction with Emotive Stories in Museums

Engaging Teens Through Playful Tangible Interaction with Emotive Stories in Museums

Learning Circuits and Coding with Arduino Board in Higher Education Using Tangible and Graphical User Interfaces

The integration of the Arduino board into educational settings has penetrated across various educational levels. The teaching of this subject can be accomplished by (a) using real components in breadboards, (b) prefabricated modular boards that snap together, and (c) utilizing computer simulations. Yet, it is unknown which interface offers a more effective learning experience. Therefore, this experimental study aims to compare the effectiveness of these interfaces in a series of three laboratory exercises involving 110 university students, who were divided into three groups: (a) the first group used a tangible user interface, implementing circuits on breadboards, (b) the second group also used a tangible interface but with modular boards, and (c) the third group used a graphical user interface to simulate circuits using Tinkercad. For each laboratory exercise, students completed both pretests and posttests. Also, they provided feedback through five Likert-type attitude questions regarding their experiences. In terms of data analysis, t-tests, ANOVA, and ANCOVA, along with bootstrapping, and principal component analysis were employed. The results suggest that among the participants, those who used a graphical user interface stated that their understanding of the interconnection of components in microcontroller circuits was enhanced, while students with previous experience in microcontroller labs found the circuit creation process easier than students without experience.

Passive Haptics and Conversational Avatars for Interacting with Ancient Egypt Remains in High-Fidelity Virtual Reality Experiences

As extended reality continues to grow, new possibilities arise to provide users with novel ways to experience cultural heritage (CH). In particular, applications based on virtual reality (VR), such as virtual museums, have gained increasing popularity, since they can offer new ways for preserving and presenting CH content that are not feasible in physical museums. Despite the numerous benefits, the level of immersion and presence provided by VR experiences still present challenges that could hinder the effectiveness of this technology in the CH context. In this perspective, it is crucial to provide the users with high-fidelity experiences, in which also the interaction with the objects and the characters populating virtual environments are realistic and natural. This article focuses on this challenge and specifically investigates how the combined use of tangible and speech interfaces can help improve the overall experience. To this aim, a immersive VR experience is proposed, which allows the users to manipulate virtual objects belonging to a museum collection (in the specific case, Ancient Egypt remains) by physically operating on 3D printed replicas and to talk with a curator’s avatar to get explanations by using their voice. A user study was conducted to evaluate the impact of the considered interfaces on immersion, presence, user experience, usability, and intention to visit, comparing the richest configuration against simpler setups obtained by either removing the tangible interface, the speech interface, or both (and using only handheld controllers). The results show that the combined use of the two interfaces can effectively contribute at making the CH experience in VR more engaging.

The FlexiBoard: Tangible and Tactile Graphics for People with Vision Impairments

Over the last decade, several projects have demonstrated how interactive tactile graphics and tangible interfaces can improve and enrich access to information for people with vision impairments. While the former can be used to display a relatively large amount of information, they cannot be physically updated, which constrains the type of tasks that they can support. On the other hand, tangible interfaces are particularly suited for the (re)construction and manipulation of graphics, but the use of physical objects also restricts the type and amount of information that they can convey. We propose to bridge the gap between these two approaches by investigating the potential of tactile and tangible graphics for people with vision impairments. Working closely with special education teachers, we designed and developed the FlexiBoard, an affordable and portable system that enhances traditional tactile graphics with tangible interaction. In this paper, we report on the successive design steps that enabled us to identify and consider technical and design requirements. We thereafter explore two domains of application for the FlexiBoard: education and board games. Firstly, we report on one brainstorming session that we organized with four teachers in order to explore the application space of tangible and tactile graphics for educational activities. Secondly, we describe how the FlexiBoard enabled the successful adaptation of one visual board game into a multimodal accessible game that supports collaboration between sighted, low-vision and blind players.

Unmasking the Power of Play Through TUI Designs

Research on the potential benefits of technology for autistic children is an emergent field in Human-Computer Interaction (HCI), especially within the Child-Computer Interaction Community. At the same time, there are concerns about what these interventions and technologies are for and who benefits. We present a research and design approach for Tangible User Interfaces (TUIs) for minimally verbal to nonverbal autistic children following a neurodiversity narrative through three field studies developed and evaluated with three groups of children within a semi-structured scholastic environment between 2018 and 2021 in the UK. We discuss our insights for research and TUI designs in the context of social play for nonverbal autistic children and critically reflect on the methods and approaches we used. We do this to disrupt the normalisation agenda that subtly permeates the field of HCI and to direct designers’ attention toward supporting autistic ways of being in the world.

An Interactive Hybrid Book Integrating Capacitive, Piezoelectric, and Piezoresistive Polymer‐Based Technologies

The arising of novel paradigms in human–computer interaction, including tangible user interfaces, shape‐changing interfaces, or ubiquitous computing, is breaking down the boundary between physical and digital realms, leading to more natural and immersive forms of interaction. Advances in active and responsive smart materials allow to transform once inanimate objects, such as books, into actively responsive objects. Herein, a hybrid interactive book has been developed. It includes a capacitive touchpad, a piezoelectric speaker, and a piezoresistive platform for drawing and writing digitalization. All interactive elements are based on electroactive polymer films and are processed by additive manufacturing techniques. This work provides a compelling demonstration of how advanced materials can be seamlessly integrated in traditional physical interfaces to enhance interactivity and user experience.

Integrating Generative Artificial Intelligence to a Project-Based Tangible Interaction Course

Integrating Generative Artificial Intelligence to a Project-Based Tangible Interaction Course