Formal User Study Research Articles

Enhancing augmented reality with machine learning for hands-on origami training

This research explores integrating augmented reality (AR) with machine learning (ML) to enhance hands-on skill acquisition through origami folding. We developed an AR system using the YOLOv8 model to provide real-time feedback and automatic validation of each folding step, offering step-by-step guidance to users. A novel approach to training dataset preparation was introduced, which improves the accuracy of detecting and assessing origami folding stages. In a formative user study involving 16 participants tasked with folding multiple origami models, the results revealed that while the ML-driven feedback increased task completion times, it also made participants feel more confident throughout the folding process. However, they also reported that the feedback system added cognitive load, slowing their progress, though it provided valuable guidance. These findings suggest that while ML-supported AR systems can enhance the user experience, further optimization is required to streamline the feedback process and improve efficiency in complex manual tasks.

TangibleTale: Designing Tangible Child–Parent Interactive Storytelling for Promoting Eating Behaviors

Tangible narrative allow children to interact with physical objects and provides an immersive storytelling approach to influence their cognition and behavior. However, research on using tangible narratives to encourage behaviors in children remains limited. To improve children’s eating behaviors, we combined narrative transportation theory with behavior change principles to conduct a design study, called TangibleTale. Specifically, we conducted a formative user study (N = 12 pairs) to identify the characteristics of children’s engagement with tangible narrative and their interactions with parents, which were then incorporated into a design workshop (N = 12) to develop an interactive product comprising tangible elements and an accompanying app. With the produced outcomes, we conducted a comparative experiment (N = 24 pairs) in a home setting to verify and explore the role of tangible narrative in child–parent mealtime interaction. Finally, we formulated design guidelines for a tangible narrative that can serve as a reference to assist in creating more impactful products that foster positive behavioral growth in children.

MINARO DRS: usability study of a robotic-assisted laminectomy.

Although the literature shows that robotic assistance can support the surgeon, robotic systems are not widely spread in clinics. They often incorporate large robotic arms adopted from the manufacturing industry, imposing safety hazards when in contact with the patient or surgical staff. We approached this limitation with a modular dual robot consisting of an ultra-lightweight carrier robot for rough prepositioning and small, highly dynamic, application-specific, interchangeable tooling robots. A formative usability study with N = 10 neurosurgeons was conducted using a prototype of a novel tooling robot for laminectomy to evaluate the system's usability. The participants were asked to perform three experiments using the robotic system: (1) prepositioning with the carrier robot and milling into (2) a block phantom as well as (3) a spine model. All neurosurgeons could perform a simulated laminectomy on a spine phantom using the robotic system. On average, they rated the usability of this first prototype already between good and excellent (SUS-Score above 75%). Eight out of the ten participants preferred robotic-assisted milling over manual milling. For prepositioning, the developed haptic guidance showed significantly higher effectiveness and efficiency than visual navigation. The proposed dual robot system showed the potential to increase safety in the operating room because of the synergistic hands-on control and the ultra-lightweight design of the carrier robot. The modular design allows for easy adaptation to various surgical procedures. However, improvements are needed in the ergonomics of the tooling robot and the complexity of the virtual fixtures. The cooperative dual robot system can subsequently be tested in a cadaver laboratory and in vivo on animals.

A Formative Usability Study Evaluating a Decision Aid for Caregivers of Incapacitated Patients

Families making medical decisions for an incapacitated loved one need to process medical information for various care pathways while balancing different perspectives and experiencing distress. A decision aid tool to help family members make medical decisions on the patient’s behalf should be easy to use and not create additional burdens. A formative usability study with high, medium, and low verbosity versions of an initial decision aid was conducted with university students before clinical testing. The results showed no significant differences in usability metrics between different verbosity levels, but the qualitative findings indicated areas of improvement in the organization of information that could lead to improved usability of the decision aid. While the findings are limited due to the study being void of some of the burdens families face in true clinical situations, the study allowed us to identify numerous usability issues before testing with target users.

ViRgilites: Multilevel Feedforward for Multimodal Interaction in VR

Navigating the interaction landscape of Virtual Reality (VR) and Augmented Reality (AR) presents significant complexities due to the plethora of available input hardware and interaction modalities, compounded by spatially diverse visual interfaces. Such complexities elevate the likelihood of user errors, necessitating frequent backtracking. To address this, we introduce ViRgilites, a virtual guidance framework that delivers multi-level feedforward information covering the available interaction techniques as well as the future possibilities to interact with virtual objects, anticipating the interaction effects and how they fit with the overall user's goal. ViRgilites is engineered to facilitate task execution, empowering users to make informed decisions about action methodologies and alternative courses of action. This paper presents the architecture and functionality of ViRgilites and demonstrates its efficacy through evaluation with a formative user study.

SPROUT: an Interactive Authoring Tool for Generating Programming Tutorials with the Visualization of Large Language Models.

The rapid development of large language models (LLMs), such as ChatGPT, has revolutionized the efficiency of creating programming tutorials. LLMs can be instructed with text prompts to generate comprehensive text descriptions for code snippets provided by users. However, the lack of transparency in the end-to-end generation process has hindered the understanding of model behavior and limited user control over the generated results. To tackle this challenge, we introduce a novel approach that breaks down the programming tutorial creation task into actionable steps. By employing the tree-of-thought method, LLMs engage in an exploratory process to generate diverse and faithful programming tutorials. We then present SPROUT, an authoring tool equipped with a series of interactive visualizations that empower users to have greater control and understanding of the programming tutorial creation process. A formal user study demonstrated the effectiveness of SPROUT, showing that our tool assists users to actively participate in the programming tutorial creation process, leading to more reliable and customizable results. By providing users with greater control and understanding, SPROUT enhances the user experience and improves the overall quality of programming tutorial.A free copy of this paper and all supplemental materials are available at https://osf.io/uez2t/?view only = 5102e958802341daa414707646428f86.

#2978 Formative usability test of a novel device for peritoneal dialysis (WEAKID)

Abstract Background and Aims Chronic kidney disease is a growing problem worldwide with a growing number of patients requiring kidney replacement therapy. In recent years we have developed a novel device for peritoneal dialysis (PD), the WEarable Artificial KIDney (WEAKID); composed of a wearable device connected to a dialysate reservoir. WEAKID uses both sorbent technology and continuous flow PD to improve dialysis efficacy. Formative usability testing aids to the identification of design issues that may interfere with a device's safe and effective use and thus may influence decisions made in the design process. Here, we present the results of an international, multicentre, formative usability study of WEAKID among nurses and patients. Method This study used a mixed-methods design and was conducted in hospitals in Italy (Modena University Hospital), Spain (La Paz University Hospital, Madrid), and the Netherlands (University Medical Center Utrecht) between November 2021 and December 2022. An individual, formative usability testing session was conducted with each participant (nurse or patient) under the conditions of simulated use. Participants were asked to perform basic operational procedures (i.e. turning the device on/off, input of dialysis settings, handling an alarm) with the device according to a usability task list. Their performance was observed and use errors (i.e., something that a participant does or fails to do that results in an unexpected or unintended outcome) were noted. Furthermore, participants were asked for feedback, both verbally and by means of completing the system usability scale (SUS), reflecting perceived usability on a scale from 0 (worst) to 100 (best). Results We included 14 participants (n = 6 nurses, n = 8 patients) in three participating centres. All nurses were female and had a median experience with PD of 20.5 years (range 0.5-32). Patients were generally highly educated (n = 5, 63%). In total, 22 use errors occurred, 10 in nurses and 12 in patients. These use errors included factors not related to the device, e.g. not taking appropriate hygiene measures. In particular, use errors occurred during the priming/connection of catheter and handling the alarm. Participants’ feedback mainly concerned the small screen (and thus font) size, user interface (e.g. lack of colour-coded soft buttons), and (short) length of the patient line. The mean SUS score was 70 (range 57.5–85) for nurses, and 71.9 (range 55–87.5) for patients. Conclusion Despite the slightly above-average perceived usability of the device's current design stage, participants made a lot of suggestions for improvements, and usability must be improved to make the device ready for market introduction.

An explanation-oriented inquiry dialogue game for expert collaborative recommendations

This work presents a requirement analysis for collaborative dialogues among medical experts and an inquiry dialogue game based on this analysis for incorporating explainability into multiagent system design. The game allows experts with different knowledge bases to collaboratively make recommendations while generating rich traces of the reasoning process through combining explanation-based illocutionary forces in an inquiry dialogue. The dialogue game was implemented as a prototype web-application and evaluated against the specification through a formative user study. The user study confirms that the dialogue game meets the needs for collaboration among medical experts. It also provides insights on the real-life value of dialogue-based communication tools for the medical community.

Stylus and Gesture Asymmetric Interaction for Fast and Precise Sketching in Virtual Reality

This research investigates fast and precise Virtual Reality (VR) sketching methods with different tool-based asymmetric interfaces. In traditional real-world drawing, artists commonly employ an asymmetric interaction system where each hand holds different tools, facilitating diverse and nuanced artistic expressions. However, in virtual reality (VR), users are typically limited to using identical tools in both hands for drawing. To bridge this gap, we aim to introduce specifically designed tools in VR that replicate the varied tool configurations found in the real world. Hence, we developed a VR sketching system supporting three hybrid input techniques using a standard VR controller, a VR stylus, or a data glove. We conducted a formal user study consisting of an internal comparative experiment with four conditions and three tasks to compare three asymmetric input methods with each other and with a traditional symmetric controller-based solution based on questionnaires and performance evaluations. The results showed that in contrast to symmetric dual VR controller interfaces, the asymmetric input with gestures significantly reduced task completion times while maintaining good usability and input accuracy with a low task workload. This shows the value of asymmetric input methods for VR sketching. We also found that the overall user experience could be further improved by optimizing the tracking stability of the data glove and the VR stylus.

SurfaceCast: Ubiquitous, Cross-Device Surface Sharing

Real-time online interaction is the norm today. Tabletops and other dedicated interactive surface devices with direct input and tangible interaction can enhance remote collaboration, and open up new interaction scenarios based on mixed physical/virtual components. However, they are only available to a small subset of users, as they usually require identical bespoke hardware for every participant, are complex to setup, and need custom scenario-specific applications. We present SurfaceCast, a software toolkit designed to merge multiple distributed, heterogeneous end-user devices into a single, shared mixed-reality surface. Supported devices include regular desktop and laptop computers, tablets, and mixed-reality headsets, as well as projector-camera setups and dedicated interactive tabletop systems. This device-agnostic approach provides a fundamental building block for exploration of a far wider range of usage scenarios than previously feasible, including future clients using our provided API. In this paper, we discuss the software architecture of SurfaceCast, present a formative user study and a quantitative performance analysis of our framework, and introduce five example application scenarios which we enhance through the multi-user and multi-device features of the framework. Our results show that the hardware- and content-agnostic architecture of SurfaceCast can run on a wide variety of devices with sufficient performance and fidelity for real-time interaction.

A Generic Interactive Membership Function for Categorization of Quantities.

The membership function is to categorize quantities along with a confidence degree. This article investigates a generic user interaction based on this function for categorizing various types of quantities without modification, which empowers users to articulate uncertainty categorization and enhance their visual data analysis significantly. We present the technique design and an online prototype, supplementing with insights from three case studies that highlight the technique's efficacy among different types of quantities. Furthermore, we conduct a formal user study to scrutinize the process and reasoning users employ while utilizing our technique. The findings indicate that our technique can help users create customized categories. Both our code and the interactive prototype are made available as open-source resources, intended for application across varied domains as a generic tool.

Evolution of the RebiSmart® Electromechanical Autoinjector to Improve Usability in Support of Adherence to Subcutaneous Interferon Beta-1a Therapy for People Living with Multiple Sclerosis.

The RebiSmart® electromechanical autoinjector supports people living with relapsing multiple sclerosis (MS) with their adherence to treatment with subcutaneous interferon beta-1a (sc IFN β-1a; Rebif®), a well-established and effective therapy. We report on the validation of the next-generation device, RebiSmart3.0, tailored to meet patients' changing needs. To conclude a series of formative usability studies, a final formative study of an updated prototype version of the RebiSmart electromechanical autoinjector was conducted to identify the device's strengths, potential device-related use errors, opportunities for improvement, and to inform device safety. The findings were incorporated into the next-generation device, RebiSmart3.0, which was then evaluated in a summative usability study involving 45participants. The study consisted of evaluation activities - use scenarios and knowledge tasks - designed to validate mitigations to reduce the risks of not successfully completing critical tasks for successful administration of medication. During each evaluation activity, observations (including use errors, instances of moderator assistance, close calls, and difficulties) were recorded, focusing on the potential for serious harm arising from not completing critical tasks. Participants then provided their subjective assessment of RebiSmart3.0 as part of a user needs survey that assessed device usability and design. Regarding critical tasks, main findings were failure to inspect/dispose of the cartridge and not washing hands or disinfecting the injection site. These issues could be readily overcome by modifying future training. In the subjective assessment, 43 out of 45 participants considered the updated device safe to use as-is. In the user needs survey, overall, the participants rated the device positively. Findings validate the safety of use of the next-generation device, RebiSmart3.0, through a comprehensive evaluation of use scenarios and knowledge tasks by the study participants, who provided positive ratings of the device in the user needs survey.

Two-step techniques for accurate selection of small elements in VR environments

One of the key interactions in 3D environments is target acquisition, which can be challenging when targets are small or in cluttered scenes. Here, incorrect elements may be selected, leading to frustration and wasted time. The accuracy is further hindered by the physical act of selection itself, typically involving pressing a button. This action reduces stability, increasing the likelihood of erroneous target acquisition. We focused on molecular visualization and on the challenge of selecting atoms, rendered as small spheres. We present two techniques that improve upon previous progressive selection techniques. They facilitate the acquisition of neighbors after an initial selection, providing a more comfortable experience compared to using classical ray-based selection, particularly with occluded elements. We conducted a pilot study followed by two formal user studies. The results indicated that our approaches were highly appreciated by the participants. These techniques could be suitable for other crowded environments as well.

Augmented Reality Visualization and Interaction for COVID-19 CT-Scan NN Automated Segmentation: A Validation Study

Although medical imaging technology has persisted in evolving over the last decades, the techniques and technologies used for analytical and visualisation purposes have remained constant. Manual or semi-automatic segmentation is, in many cases, complicated. It requires the intervention of a specialist and is time-consuming, especially during the Coronavirus disease (COVID-19) pandemic, which has had devastating medical and economic consequences. Processing and visualising medical images with advanced techniques represent medical professionals’ breakthroughs. This paper studies how augmented reality (AR) and artificial intelligence (AI) can transform medical practice during COVID-19 and post-COVID-19 pandemic. Here we report an augmented reality visualisation and interaction platform; it covers the whole process from uploading chest Ct-scan images to automatic segmentation-based deep learning, 3D reconstruction, 3D visualisation, and manipulation. AR provides a more realistic 3D visualisation system, allowing doctors to effectively interact with the generated 3D model of segmented lungs and COVID-19 lesions. We use the U-Net Neural Network (NN) for automated segmentation. The statistical measures obtained using the Dice score, pixel accuracy, sensitivity, G-mean, and specificity are 0.749, 0.949, 0.956, 0.955, and 0.954, respectively. The user-friendliness and usability are objectified by a formal user study that compared our augmented reality-assisted design to the standard diagnosis setup. One hundred and six doctors and medical students, including eight senior medical lecturers, volunteered to assess our platform. The platform could be used as an aid diagnosis tool to identify and analyse the COVID-19 infectious or as a training tool for residents and medical students. The prototype can be extended to other pulmonary pathologies.

Usability Testing of a Web-Based Empathy Training Portal: Mixed Methods Study.

The prepandemic period saw a rise in web-based teaching. However, web-based tools for teaching the essential clinical skill of cognitive empathy (also known as perspective taking) remain limited. More of these tools are needed and require testing for ease of use and understanding by students. This study aimed to evaluate the usability of the In Your Shoes web-based empathy training portal application for students using quantitative and qualitative methods. This 3-phase formative usability study used a mixed methods design. In mid-2021, we conducted a remote observation of student participants interacting with our portal application. Their qualitative reflections were captured, followed by data analysis and iterative design refinements of the application. Overall, 8 third- and fourth-year nursing students from an undergraduate baccalaureate program at a Canadian university, in the western province of Manitoba, were included in this study. Participants in phases 1 and 2 were remotely observed by 3 research personnel while engaged in predefined tasks. In phase 3, two student participants were asked to use the application as they liked in their own environments, after which a video-recorded exit interview with a think-aloud process was conducted as participants responded to the System Usability Scale. We calculated descriptive statistics and performed content analysis to analyze the results. This small study included 8 students with a range of technology skills. Usability themes were based on participants' comments on the application's appearance, content, navigation, and functionality. The biggest issues that participants experienced were with navigating the application's "tagging" features during video analysis and the length of educational material. We also observed variations in 2 participants' system usability scores in phase 3. This may be because of their different comfort levels with technology; however, additional research is required. We made iterative refinements to our prototype application (eg, added pop-up messages and provided a narrated video on the application's "tagging" function) based on participant feedback. With increasing engagement in web-based teaching, technology has become an essential medium for receiving health care education. We developed a novel prototype application as a supplemental classroom tool to foster students' self-directed learning of empathy. This study provided direction for refinements to optimize the usability of and satisfaction with this innovative application. Qualitative feedback revealed favorable input toward learning perspective taking place on the web and helpful recommendations for improving user experiences with the application. We could not fully assess the application's key functions owing to the COVID-19 protocols. Thus, our next step is to obtain feedback from a larger sample of student users, whose experiences performing "live" video capture, annotation, and analysis will be more authentic and wholesome with the refined application. We discuss our findings in relation to research on nursing education, perspective taking, and adaptive e-learning.

Nanotilus: Generator of Immersive Guided-Tours in Crowded 3D Environments.

Immersive virtual reality environments are gaining popularity for studying and exploring crowded three-dimensional structures. When reaching very high structural densities, the natural depiction of the scene produces impenetrable clutter and requires visibility and occlusion management strategies for exploration and orientation. Strategies developed to address the crowdedness in desktop applications, however, inhibit the feeling of immersion. They result in nonimmersive, desktop-style outside-in viewing in virtual reality. This article proposes Nanotilus-a new visibility and guidance approach for very dense environments that generates an endoscopic inside-out experience instead of outside-in viewing, preserving the immersive aspect of virtual reality. The approach consists of two novel, tightly coupled mechanisms that control scene sparsification simultaneously with camera path planning. The sparsification strategy is localized around the camera and is realized as a multi-scale, multi-shell, variety-preserving technique. When Nanotilus dives into the structures to capture internal details residing on multiple scales, it guides the camera using depth-based path planning. In addition to sparsification and path planning, we complete the tour generation with an animation controller, textual annotation, and text-to-visualization conversion. We demonstrate the generated guided tours on mesoscopic biological models - SARS-CoV-2 and HIV. We evaluate the Nanotilus experience with a baseline outside-in sparsification and navigational technique in a formal user study with 29 participants. While users can maintain a better overview using the outside-in sparsification, the study confirms our hypothesis that Nanotilus leads to stronger engagement and immersion.

Labeling Out-of-View Objects in Immersive Analytics to Support Situated Visual Searching.

Augmented Reality (AR) embeds digital information into objects of the physical world. Data can be shown in-situ, thereby enabling real-time visual comparisons and object search in real-life user tasks, such as comparing products and looking up scores in a sports game. While there have been studies on designing AR interfaces for situated information retrieval, there has only been limited research on AR object labeling for visual search tasks in the spatial environment. In this article, we identify and categorize different design aspects in AR label design and report on a formal user study on labels for out-of-view objects to support visual search tasks in AR. We design three visualization techniques for out-of-view object labeling in AR, which respectively encode the relative physical position (height-encoded), the rotational direction (angle-encoded), and the label values (value-encoded) of the objects. We further implement two traditional in-view object labeling techniques, where labels are placed either next to the respective objects (situated) or at the edge of the AR FoV (boundary). We evaluate these five different label conditions in three visual search tasks for static objects. Our study shows that out-of-view object labels are beneficial when searching for objects outside the FoV, spatial orientation, and when comparing multiple spatially sparse objects. Angle-encoded labels with directional cues of the surrounding objects have the overall best performance with the highest user satisfaction. We discuss the implications of our findings for future immersive AR interface design.

BeHere: a VR/SAR remote collaboration system based on virtual replicas sharing gesture and avatar in a procedural task.

The online version contains supplementary material available at 10.1007/s10055-023-00748-5.

Remote synchronous usability testing of public access defibrillators during social distancing in a pandemic

Public access automated external defibrillators (AEDs) represent emergency medical devices that may be used by untrained lay-persons in a life-critical event. As such their usability must be confirmed through simulation testing. In 2020 the novel coronavirus caused a global pandemic. In order to reduce the spread of the virus, many restrictions such as social distancing and travel bans were enforced. Usability testing of AEDs is typically conducted in-person, but due to these restrictions, other usability solutions must be investigated. Two studies were conducted, each with 18 participants: (1) an in-person usability study of an AED conducted in an office space, and (2) a synchronous remote usability study of the same AED conducted using video conferencing software. Key metrics associated with AED use, such as time to turn on, time to place pads and time to deliver a shock, were assessed in both studies. There was no difference in time taken to turn the AED on in the in-person study compared to the remote study, but the time to place electrode pads and to deliver a shock were significantly lower in the in-person study than in the remote study. Overall, the results of this study indicate that remote user testing of public access defibrillators may be appropriate in formative usability studies for determining understanding of the user interface.

Smart Textile Sock System for Athletes' Self-Correction during Functional Tasks: Formative Usability Evaluation.

(1) Background: The development of a lightweight, easy-to-use system that measures the foot’s plantar pressure is becoming an increasingly important area of research in physiotherapy. For further development of the smart sock system, a formative usability study was conducted, where the smart textile sock sensor system was used for self-correction during functional tasks; (2) Methods: Five athletes from the football school participated in the formative usability study. Athletes performed pre-defined functional tasks for self-correction when interacting with the smart textile sock system. Formative usability evaluation methods: effectiveness (task success rate, error rate), efficiency (time-based), satisfaction evaluated by System Usability Scale (SUS); (3) Results: Formative usability indicators: task completeness effectiveness ranged from 40% to 100% in the first- and second-stage tasks. Completed task efficiency time: Stage 1, from 4.2 s (SD 1.3) to 88.8 s (SD 19.8); Stage 2, from 7.2 s (SD 1.9) to 9.6 s (SD 2.1). Satisfaction was assessed by the SUS system user group with 76 points (SD 7.42), which indicates “good” satisfaction; (4) Conclusions: formative usability indicators showed the need for technical improvements to the smart textile sock pressure sensor system. The SUS results indicate “good” satisfaction with the smart textile sock pressure sensor system and its application.