Conducted User Studies Research Articles

Scene-aware Foveated Rendering.

We propose a new scene-aware foveated rendering method, which incorporates the scene awareness and characteristics of the human visual system into the mapping-based foveated rendering framework. First, we generate the conservative visual importance map that encodes the visual features of the scene, visual acuity, and gaze motion. Second, we construct the pixel size control map using a convolution kernel method. Third, we utilize the pixel size control map to guide the foveated rendering. At last, a temporal coherent refinement strategy is used to maintain the smooth foveated rendering for the adjacent frames. Compared to the state-of-the-art mapping-based foveated rendering methods using the same compression ratio, our method achieves smaller MSE, higher PSNR, and SSIM in the fovea, periphery, salient regions, and the whole image. We also conducted user studies, and the results proved that the perceptual quality of our method has a high visual similarity with the around truth rendered with the full resolution.

Shared storytelling with a virtual learning companion: prospects in child-AI collaboration

Advances in natural language processing techniques enhance the capabilities of conversational agents to process input text and generate fluent responses in various genre. In the educational setting, these conversational agents can function as learning companions to promote peer learning, enhance social interaction, and motivate collaborative behavior. In this paper, we investigate how children collaborate with conversational agents in the context of story construction. We conducted user studies and administered survey forms to collect children’s evaluation of their experience, captured the child-agent conversation logs, and used an observation checklist to monitor the interaction. We analyzed the logs to determine how children respond to varying roles portrayed by the agent and the dialogue moves that motivate collaborative behavior during storytelling. We correlated these with the children’s reported perception and our own observation of the interaction based on performance, collaboration and humanity attributes. Drawing on these results, we identified the roles of conversational agents that align with children’s traits and abilities as storytellers and propose dialogue strategies to facilitate collaborative storytelling. Findings from our study can inform future educational technology designers on the roles of conversational agents as collaborative learning companions who can adapt to the individual needs and preferences of the children.

Design and Development of a Smart Fidget Toy Using Blockchain Technology to Improve Health Data Control.

This study explores the integration of blockchain technology in wearable health devices through the design and development of a Smart Fidget Toy. We aimed to investigate design challenges and opportunities of blockchain-based health devices, examine the impact of blockchain integration user experience, and assess its potential to improve data control and user trust. Using an iterative user-centered design approach, we developed a mid-fidelity prototype of a physical fidget device with a blockchain-based web application. Our key contributions include the design of a fidget toy using blockchain for secure health data management, an iterative development process balancing user needs with blockchain integration challenges, and insights into user perceptions of blockchain wearables for health. We conducted user studies, including a survey (n = 28), focus group (n = 6), interactive wireframe testing (n = 7), and prototype testing (n = 10). Our study revealed high user interest (70%) in blockchain-based data control and sharing features and improved perceived security of data (90% of users) with blockchain integration. However, we also identified challenges in user understanding of blockchain concepts, necessitating additional support. Our smart contract, deployed on the Polygon zkEVM testnet, efficiently manages data storage and retrieval while maintaining user privacy. This research advances the understanding of blockchain applications in health wearables, offering valuable insights for the future development of this field.

Pose‐to‐Motion: Cross‐Domain Motion Retargeting with Pose Prior

AbstractCreating plausible motions for a diverse range of characters is a long‐standing goal in computer graphics. Current learning‐based motion synthesis methods rely on large‐scale motion datasets, which are often difficult if not impossible to acquire. On the other hand, pose data is more accessible, since static posed characters are easier to create and can even be extracted from images using recent advancements in computer vision. In this paper, we tap into this alternative data source and introduce a neural motion synthesis approach through retargeting, which generates plausible motion of various characters that only have pose data by transferring motion from one single existing motion capture dataset of another drastically different characters. Our experiments show that our method effectively combines the motion features of the source character with the pose features of the target character, and performs robustly with small or noisy pose data sets, ranging from a few artist‐created poses to noisy poses estimated directly from images. Additionally, a conducted user study indicated that a majority of participants found our retargeted motion to be more enjoyable to watch, more lifelike in appearance, and exhibiting fewer artifacts. Our code and dataset can be accessed here.

XR MUSE: An Open-Source Unity Framework for Extended Reality-Based Networked Multi-User Studies

In recent years, extended reality (XR) technologies have been increasingly used as a research tool in behavioral studies. They allow experimenters to conduct user studies in simulated environments that are both controllable and reproducible across participants. However, creating XR experiences for such studies remains challenging, particularly in networked, multi-user setups that investigate collaborative or competitive scenarios. Numerous aspects need to be implemented and coherently integrated, e.g., in terms of user interaction, environment configuration, and data synchronization. To reduce this complexity and facilitate development, we present the open-source Unity framework XR MUSE for devising user studies in shared virtual environments. The framework provides various ready-to-use components and sample scenes that researchers can easily customize and adapt to their specific needs.

Holographic Parallax Improves 3D Perceptual Realism

Holographic near-eye displays are a promising technology to solve long-standing challenges in virtual and augmented reality display systems. Over the last few years, many different computer-generated holography (CGH) algorithms have been proposed that are supervised by different types of target content, such as 2.5D RGB-depth maps, 3D focal stacks, and 4D light fields. It is unclear, however, what the perceptual implications are of the choice of algorithm and target content type. In this work, we build a perceptual testbed of a full-color, high-quality holographic near-eye display. Under natural viewing conditions, we examine the effects of various CGH supervision formats and conduct user studies to assess their perceptual impacts on 3D realism. Our results indicate that CGH algorithms designed for specific viewpoints exhibit noticeable deficiencies in achieving 3D realism. In contrast, holograms incorporating parallax cues consistently outperform other formats across different viewing conditions, including the center of the eyebox. This finding is particularly interesting and suggests that the inclusion of parallax cues in CGH rendering plays a crucial role in enhancing the overall quality of the holographic experience. This work represents an initial stride towards delivering a perceptually realistic 3D experience with holographic near-eye displays.

Lessons Learned from Investigating Robotics-Based, Human-like Testing of an Upper-Body Exoskeleton

Assistive devices like exoskeletons undergo extensive testing not least because of their close interaction with humans. Conducting user studies is a time-consuming process that demands expert knowledge, and it is accompanied by challenges such as low repeatability and a potential lack of comparability between studies. Obtaining objective feedback on the exoskeleton’s performance is crucial for developers and manufacturers to iteratively improve the design and development process. This paper contributes to the concept of using robots for objective exoskeleton testing by presenting various approaches to a robotic-based testing platform for upper-body exoskeletons. We outline the necessary requirements for realistically simulating use cases and evaluate different approaches using standard manipulators as robotic motion generators. Three approaches are investigated: (i) Exploiting the anthropomorphic structure of the robotic arm and directly placing it into the exoskeleton. (ii) Utilizing a customized, direct attachment between the robot and exoskeleton. (iii) Attaching a human arm dummy to the robot end effector to simulate a more realistic interface with the exoskeleton. Subsequently, we discuss and compare the results against the aforementioned requirements of a systematic testing platform. Our conclusion emphasizes that achieving objective and realistic testing necessitates highly specialized hardware, algorithms, and further research to address challenging requirements.

Towards Development of a Multilingual Mobile Chat Application for Enhanced Global Communication

The advent of mobile chat applications has revolutionized everyday communication. These applications facilitate the exchange of user's textual and multimedia content across languages and cultures. Most chat applications are known to only support a limited set of predominantly spoken languages, thereby, leaving a substantial portion of the user population without adequate multilingual support. This paper aims to bridge the linguistic gap by presenting Kobapp, a multilingual chat application. The Kobapp, leverages some of the cutting-edge technologies, such as React-Native, Next.js, and the DeepL API, to offer real-time, accurate translations while at the same time offering user privacy. The development process of the Kobapp is outlined from the system architecture and design, emphasizing the integration of a client-side (Android) and server-side using Node.js, Express.js, and MongoDB. Notably, user feedback plays a crucial role in shaping an application's features and functionality. Therefore, the application’s performance was evaluated through a conducted user study. Results of the study indicate a strong positive linear relationship between overall user satisfaction and translation accuracy for different language pairs. Moreover, the absence of outliers and the model's significance further reinforces the application's commitment to data quality and accuracy. Future research will explore new dimensions in multilingual communication and applications to promote a truly global community.

Meetor: A Human-Centered Automatic Video Editing System for Meeting Recordings

Widely adopted digital cameras and smartphones have generated a large number of videos, which have brought a tremendous workload to video editors. Recently, a variety of automatic/semi-automatic video editing methods have been proposed to tackle these issues in some specific areas. However, for the production of meeting recordings, the existing studies highly depend on extra equipment in the conference venues, such as the infrared camera or special microphone, which are not practical. In this paper, we design and implement Meetor, a human-centered automatic video editing system for meeting recordings. The Meetor mainly contains three parts: an audio-based video synchronization algorithm, human-centered video content flaw detection algorithms, and an automatic video editing algorithm. Two main experiments are conducted from both objective and subjective aspects to evaluate the performance of the Meetor. The experimental results on a testbed illustrate that the proposed algorithms could achieve state-of-the-art (SOTA) performance in video content flaw detection. On the other hand, the conducted user study demonstrates that Meetor could generate meeting recordings with a satisfactory quality compared with professional video editors. Moreover, we also present a practical application of the Meetor in a university campus prototype, in which the Meetor is applied in the automatic editing of lecture recordings. All in all, the proposed Meetor can be utilized in practical applications to release the workload of professional video editors.

Implementing Sign Language Recognition System using Flex Sensors

Sign languages rely on a combination of handshapes, facial expressions, and body movements to convey meaning. They are usually learnt at a tender age as one’s first language. This paper is aimed at designing, implementing, and developing a sensor-based smart glove system for the recognition of sign language. The system includes a wearable glove embedded with flex sensors for detecting hand movement and gestures used in sign language. This information from the sensors is fed into a microcontroller running an algorithm that identifies the signs and modulates them into speech. The system was aimed to provide an inexpensive and effective solution for the deaf/dumb community to communicate with others through sign language. Thirty-one phrases have been successfully obtained in the implemented system. The evaluation regarding the performance of the system was conducted by conducting user studies and tests, and results were presented to show the effectiveness of the proposed solution.

SNIL: Generating Sports News From Insights With Large Language Models.

To enhance the appeal and informativeness of data news, there is an increasing reliance on data analysis techniques and visualizations, which poses a high demand for journalists' abilities. While numerous visual analytics systems have been developed for deriving insights, few tools specifically support and disseminate viewpoints for journalism. Thus, this work aims to facilitate the automatic creation of sports news from natural language insights. To achieve this, we conducted an extensive preliminary study on the published sports articles. Based on our findings, we propose a workflow - 1) exploring the data space behind insights, 2) generating narrative structures, 3) progressively generating each episode, and 4) mapping data spaces into communicative visualizations. We have implemented a human-AI interaction system called SNIL, which incorporates user input in conjunction with large language models (LLMs). It supports the modification of textual and graphical content within the episode-based structure by adjusting the description. We conduct user studies to demonstrate the usability of SNIL and the benefit of bridging the gap between analysis tasks and communicative tasks through expert and fan feedback.

Reconstructing Image Composition: Computation of Leading Lines.

The composition of an image is a critical element chosen by the author to construct an image that conveys a narrative and related emotions. Other key elements include framing, lighting, and colors. Assessing classical and simple composition rules in an image, such as the well-known "rule of thirds", has proven effective in evaluating the aesthetic quality of an image. It is widely acknowledged that composition is emphasized by the presence of leading lines. While these leading lines may not be explicitly visible in the image, they connect key points within the image and can also serve as boundaries between different areas of the image. For instance, the boundary between the sky and the ground can be considered a leading line in the image. Making the image's composition explicit through a set of leading lines is valuable when analyzing an image or assisting in photography. To the best of our knowledge, no computational method has been proposed to trace image leading lines. We conducted user studies to assess the agreement among image experts when requesting them to draw leading lines on images. According to these studies, which demonstrate that experts concur in identifying leading lines, this paper introduces a fully automatic computational method for recovering the leading lines that underlie the image's composition. Our method consists of two steps: firstly, based on feature detection, potential weighted leading lines are established; secondly, these weighted leading lines are grouped to generate the leading lines of the image. We evaluate our method through both subjective and objective studies, and we propose an objective metric to compare two sets of leading lines.

Aspect-level Information Discrepancies across Heterogeneous Vulnerability Reports: Severity, Types and Detection Methods

Vulnerable third-party libraries pose significant threats to software applications that reuse these libraries. At an industry scale of reuse, manual analysis of third-party library vulnerabilities can be easily overwhelmed by the sheer number of vulnerabilities continually collected from diverse sources for thousands of reused libraries. Our study of four large-scale, actively maintained vulnerability databases (NVD, IBM X-Force, ExploitDB, and Openwall) reveals the wide presence of information discrepancies, in terms of seven vulnerability aspects, i.e., product, version, component, vulnerability type, root cause, attack vector, and impact, between the reports for the same vulnerability from heterogeneous sources. It would be beneficial to integrate and cross-validate multi-source vulnerability information, but it demands automatic aspect extraction and aspect discrepancy detection. In this work, we experimented with a wide range of NLP methods to extract named entities (e.g., product) and free-form phrases (e.g., root cause) from textual vulnerability reports and to detect semantically different aspect mentions between the reports. Our experiments confirm the feasibility of applying NLP methods to automate aspect-level vulnerability analysis and identify the need for domain customization of general NLP methods. Based on our findings, we propose a discrepancy-aware, aspect-level vulnerability knowledge graph and a KG-based web portal that integrates diversified vulnerability key aspect information from heterogeneous vulnerability databases. Our conducted user study proves the usefulness of our web portal. Our study opens the door to new types of vulnerability integration and management, such as vulnerability portraits of a product and explainable prediction of silent vulnerabilities.

Connect Brain, a Mobile App for Studying Depth Perception in Angiography Visualization: Gamification Study

Background One of the bottlenecks of visualization research is the lack of volunteers for studies that evaluate new methods and paradigms. The increased availability of web-based marketplaces, combined with the possibility of implementing volume rendering, a computationally expensive method, on mobile devices, has opened the door for using gamification in the context of medical image visualization studies. Objective We aimed to describe a gamified study that we conducted with the goal of comparing several cerebrovascular visualization techniques and to evaluate whether gamification is a valid paradigm for conducting user studies in the domain of medical imaging. Methods The study was implemented in the form of a mobile game, Connect Brain, which was developed and distributed on both Android (Google LLC) and iOS (Apple Inc) platforms. Connect Brain features 2 minigames: one asks the player to make decisions about the depth of different vessels, and the other asks the player to determine whether 2 vessels are connected. Results The gamification paradigm, which allowed us to collect many data samples (5267 and 1810 for the depth comparison and vessel connectivity tasks, respectively) from many participants (N=111), yielded similar results regarding the effectiveness of visualization techniques to those of smaller in-laboratory studies. Conclusions The results of our study suggest that the gamification paradigm not only is a viable alternative to traditional in-laboratory user studies but could also present some advantages.

Why Hard Code the Bionic Limbs When They Can Learn From Humans?

In this paper, we propose a task-generic learning-based model for the control of a powered ankle exoskeleton. In contrast to the traditional state machine-based control approaches that hard codes the transition heuristics for the different states and motion conditions during gait, we propose to learn the finer constraints of gait from multiple demonstrations of human gait. We validate our proposed approach on a dataset of ten subjects walking on various inclines and at multiple speeds. We deploy our model on an ankle exoskeleton, and conduct user studies on able-bodied subjects who perform gait scenarios across varying speeds and inclines. We conduct multiple online experiments to validate our learning-based approach for different motion conditions, e.g., normal walking, walking at different speeds and inclines, turns, cross-overs with variable speed and cadence, walking on a treadmill as well as on level ground. We find that our proposed learning-based model has the capability to extrapolate its learned decision rules to support untrained gait conditions, for, e.g., walking at higher speeds and inclines not seen during training. The subjects were able to adapt to the different gait scenarios comfortably without loss of stability.

Towards Attention–aware Foveated Rendering

Foveated graphics is a promising approach to solving the bandwidth challenges of immersive virtual and augmented reality displays by exploiting the falloff in spatial acuity in the periphery of the visual field. However, the perceptual models used in these applications neglect the effects of higherlevel cognitive processing, namely the allocation of visual attention, and are thus overestimating sensitivity in the periphery in many scenarios. Here, we introduce the first attention-aware model of contrast sensitivity. We conduct user studies to measure contrast sensitivity under different attention distributions and show that sensitivity in the periphery drops significantly when the user is required to allocate attention to the fovea. We motivate the development of future foveation models with another user study and demonstrate that tolerance for foveation in the periphery is significantly higher when the user is concentrating on a task in the fovea. Analysis of our model predicts significant bandwidth savings over those afforded by current models. As such, our work forms the foundation for attention-aware foveated graphics techniques.

What’s (Not) Working in Programmer User Studies?

A key goal of software engineering research is to improve the environments, tools, languages, and techniques programmers use to efficiently create quality software. Successfully designing these tools and demonstrating their effectiveness involves engaging with tool users—software engineers. Researchers often want to conduct user studies of software engineers to collect direct evidence. However, running user studies can be difficult, and researchers may lack solution strategies to overcome the barriers, so they may avoid user studies. To understand the challenges researchers face when conducting programmer user studies, we interviewed 26 researchers. Based on the analysis of interview data, we contribute (i) a taxonomy of 18 barriers researchers encounter; (ii) 23 solution strategies some researchers use to address 8 of the 18 barriers in their own studies; and (iii) 4 design ideas, which we adapted from the behavioral science community, that may lower 8 additional barriers. To validate the design ideas, we held an in-person all-day focus group with 16 researchers.

Online view enhancement for exploration inside medical volumetric data using virtual reality

Background and Objective:Medical image visualization is an essential tool for conveying anatomical information. Ray-casting-based volume rendering is commonly used for generating visualizations of raw medical images. However, exposing a target area inside the skin often requires manual tuning of transfer functions or segmentation of original images, as preset parameters in volume rendering may not work well for arbitrary scanned data. This process is tedious and unnatural. To address this issue, we propose a volume visualization system that enhances the view inside the skin, enabling flexible exploration of medical volumetric data using virtual reality. Methods:In our proposed system, we design a virtual reality interface that allows users to walk inside the data. We introduce a view-dependent occlusion weakening method based on geodesic distance transform to support this interaction. By combining these methods, we develop a virtual reality system with intuitive interactions, facilitating online view enhancement for medical data exploration and annotation inside the volume. Results:Our rendering results demonstrate that the proposed occlusion weakening method effectively weakens obstacles while preserving the target area. Furthermore, comparative analysis with other alternative solutions highlights the advantages of our method in virtual reality. We conducted user studies to evaluate our system, including area annotation and line drawing tasks. The results showed that our method with enhanced views achieved 47.73% and 35.29% higher accuracy compared to the group with traditional volume rendering. Additionally, subjective feedback from medical experts further supported the effectiveness of the designed interactions in virtual reality. Conclusions:We successfully address the occlusion problems in the exploration of medical volumetric data within a virtual reality environment. Our system allows for flexible integration of scanned medical volumes without requiring extensive manual preprocessing. The results of our user studies demonstrate the feasibility and effectiveness of walk-in interaction for medical data exploration.

Developing a Multimodal Classroom Engagement Analysis Dashboard for Higher-Education

Developing learning analytics dashboards (LADs) is a growing research interest as online learning tools have become more accessible in K-12 and higher education settings. This paper reports our multimodal classroom engagement data analysis and dashboard design process and the resulting engagement dashboard. Our work stems from the importance of monitoring classroom engagement, which refers to students' active physical and cognitive involvement in learning that influences their motivation and success in a given course. To monitor this vital facade of learning, we developed an engagement dashboard using an iterative and user-centered process. We first created a multimodal machine learning model that utilizes face and pose features obtained from recent deep learning models. Then, we created a dashboard where users can view their engagement over time and discover their learning/teaching patterns. Finally, we conducted user studies with undergraduate and graduate-level participants to obtain feedback on our dashboard design. Our paper makes three contributions by (1) presenting a student-centric, open-source dashboard, (2) demonstrating a baseline architecture for engagement analysis using our open-access data, and (3) presenting user insights and design takeaways to inspire future LADs. We expect our research to guide the development of tools for novice teacher education, student self-evaluation, and engagement evaluation in crowded classrooms.

FABRIC: A Framework for the Design and Evaluation of Collaborative Robots with Extended Human Adaptation

A limitation for collaborative robots (cobots) is their lack of ability to adapt to human partners, who typically exhibit an immense diversity of behaviors. We present an autonomous framework as a cobot’s real-time decision-making mechanism to anticipate a variety of human characteristics and behaviors, including human errors, toward a personalized collaboration. Our framework handles such behaviors in two levels: (1) short-term human behaviors are adapted through our novel Anticipatory Partially Observable Markov Decision Process (A-POMDP) models, covering a human’s changing intent (motivation), availability, and capability; (2) long-term changing human characteristics are adapted by our novel Adaptive Bayesian Policy Selection (ABPS) mechanism that selects a short-term decision model, e.g., an A-POMDP, according to an estimate of a human’s workplace characteristics, such as her expertise and collaboration preferences. To design and evaluate our framework over a diversity of human behaviors, we propose a pipeline where we first train and rigorously test the framework in simulation over novel human models. Then, we deploy and evaluate it on our novel physical experiment setup that induces cognitive load on humans to observe their dynamic behaviors, including their mistakes, and their changing characteristics such as their expertise. We conduct user studies and show that our framework effectively collaborates non-stop for hours and adapts to various changing human behaviors and characteristics in real-time. That increases the efficiency and naturalness of the collaboration with a higher perceived collaboration, positive teammate traits, and human trust. We believe that such an extended human-adaptation is a key to the long-term use of cobots.