User Task Performance Research Articles

Do better search engines really equate to better clinical decisions? If not, why not?

AbstractPrevious research has found that improved search engine effectiveness—evaluated using a batch‐style approach—does not always translate to significant improvements in user task performance; however, these prior studies focused on simple recall and precision‐based search tasks. We investigated the same relationship, but for realistic, complex search tasks required in clinical decision making. One hundred and nine clinicians and final year medical students answered 16 clinical questions. Although the search engine did improve answer accuracy by 20 percentage points, there was no significant difference when participants used a more effective, state‐of‐the‐art search engine. We also found that the search engine effectiveness difference, identified in the lab, was diminished by around 70% when the search engines were used with real users. Despite the aid of the search engine, half of the clinical questions were answered incorrectly. We further identified the relative contribution of search engine effectiveness to the overall end task success. We found that the ability to interpret documents correctly was a much more important factor impacting task success. If these findings are representative, information retrieval research may need to reorient its emphasis towards helping users to better understand information, rather than just finding it for them.

DARPA's Explainable Artificial Intelligence Program

Dramatic success in machine learning has led to a new wave of AI applications (for example, transportation, security, medicine, finance, defense) that offer tremendous benefits but cannot explain their decisions and actions to human users. DARPA's explainable artificial intelligence (XAI) program endeavors to create AI systems whose learned models and decisions can be understood and appropriately trusted by end users. Realizing this goal requires methods for learning more explainable models, designing effective explanation interfaces, and understanding the psychologic requirements for effective explanations. The XAI developer teams are addressing the first two challenges by creating ML techniques and developing principles, strategies, and human‐computer interaction techniques for generating effective explanations. Another XAI team is addressing the third challenge by summarizing, extending, and applying psychologic theories of explanation to help the XAI evaluator define a suitable evaluation framework, which the developer teams will use to test their systems. The XAI teams completed the first of this 4‐year program in May 2018. In a series of ongoing evaluations, the developer teams are assessing how well their XAM systems' explanations improve user understanding, user trust, and user task performance.

Multi-Window 3D Interaction for Collaborative Virtual Reality.

We present a novel collaborative virtual reality system that offers multiple immersive 3D views at large 3D scenes. The physical setup consists of two synchronized multi-user 3D displays: a tabletop and a large vertical projection screen. These displays afford different presentations of the shared 3D scene. The wall display lends itself to the egocentric exploration at 1:1 scale, while the tabletop affords an allocentric overview. Additionally, handheld 3D portals facilitate the personal exploration of the scene, the comparison of views, and the exchange with others. Our developments enable seamless 3D interaction across these independent 3D views. This requires the simultaneous representation of user input in the different viewing contexts. However, the resulting interactions cannot be executed independently. The application must coordinate the interactions and resolve potential ambiguities to provide plausible effects. We analyze and document the challenges of seamless 3D interaction across multiple independent viewing windows, propose a high-level software design to realize the necessary functionality, and apply the design to a set of interaction tools. Our setup was tested in a formal user study, which revealed general advantages of collaborative 3D data exploration with multiple views in terms of user preference, comfort, and task performance.

Improving User Performance in Haptics-Based Rehabilitation Exercises by Colocation of User's Visual and Motor Axes via a Three-Dimensional Augmented-Reality Display

Serious games are recently becoming a common sight in rehabilitation settings to provide motivation for patients undergoing therapy to regain upper limb function after disability. These are often presented using a two-dimensional (2-D) monitor to the patient who uses a robotic device (haptic user interface) as the game controller. In this letter, we develop a 3-D spatial augmented reality (AR) display to colocate visual and haptic feedback to the user in three rehabilitative games. The same games are also displayed in a 2-D nonimmersive virtual reality (VR) and are compared against their AR counterpart in terms of user task performance to evaluate the benefit of the 3-D AR system. To simulate a rehabilitation scenario, able-bodied participants are put under cognitive load (CL) for simulating disability-induced cognitive deficiencies when performing the tasks. A within-subjects analysis of ten participants was carried out for the rehabilitative games. The results show that AR leads to the best user performance with or without cognitive loading. This result is most evident in dynamic exercises where the participants are required to have quick reaction times and fast movement. Furthermore, even while AR had a significant difference over VR, one of the tasks showed that the performance in AR between non-CL and CL cases was similar, thereby showing how AR can alleviate the negative effects of CL.

보급서비스품질 지각이 사용자만족과 과업성과에 미치는 영향

보급서비스품질 지각이 사용자만족과 과업성과에 미치는 영향

Keynote Speakers

Keynote Speakers

Evaluating the effectiveness of spatial augmented reality in smart manufacturing: a solution for manual working stations

Augmented reality (AR) is a key technology for the development of smart manufacturing. One of the main advantages of AR is that it can help workers to accomplish several tasks, making it possible the shift from mass production to mass customization. However, it is still not clear how these promises can be fulfilled in an industrial scenario. In particular, the question about which display solutions fit better the industrial constraints remains open. Based on the literature overview, laboratory experiments, and feedbacks from industrial companies, we supported the use of spatial augmented reality (SAR), designing a prototype intended to be used for manual working stations of the future smart factories. This work presents the evaluation of the effectiveness of conveying technical instructions with this SAR prototype as compared to paper manual. We run a within-subjects experiment with 16 participants to measure user task performance (completion times and error rates) and to collect subjective evaluation. We projected technical information on a motorbike engine during a seven-task maintenance procedure. Our results proved that SAR technology improves the operators’ performance with respect to a paper manual and that users well accept it. We found that SAR is more effective for difficult tasks than for simple ones and that the main advantage of SAR is related more to the reduction of error rates than to completion times. These results confirm the goodness of our design choices; then our prototype can be a valid candidate solution for a smart manufacturing application.

Preliminary testing by adults of a haptics-assisted robot platform designed for children with physical impairments to access play

ABSTRACTDevelopment of children’s cognitive and perceptual skills depends heavily on object exploration and experience in their physical world. For children who have severe physical impairments, one of the biggest concerns is the loss of opportunities for meaningful play with objects, including physical contact and manipulation. Assistive robots can enable children to perform object manipulation through the control of simple interfaces. Touch sensations conveyed through haptic interfaces in the form of force reflection or force assistance can help a child to sense the environment and to control a robot. A robotic system with forbidden region virtual fixtures (VFs) was tested in an object sorting task. Three sorting tasks—by color, by shape, and by both color and shape—were performed by 10 adults without disability and one adult with cerebral palsy. Tasks performed with VFs were accomplished faster than tasks performed without VFs, and deviations of the motion area were smaller with VFs than without VFs. For the participant with physical impairments, two out of three tasks were slower with the VFs. This implies that forbidden region VFs are not always able to improve user task performance. Alignment with an individual’s unique motion characteristics can improve VF assistance.

FPAA-Based Control of Bilateral Teleoperation Systems for Enhanced User Task Performance

In a bilateral teleoperation system, discrete-time implementation of the controller can cause performance degradation. This is due to a well-known stability-imposed upper bound on the product of the discrete-time controller's gain and the sampling period. In this article, for a bilateral teleoperation system, a continuous-time controller based on a Field Programmable Analog Array (FPAA) is deployed and compared in terms of performance with its discrete-time counterpart. Experimental results show that, unlike the discrete-time controller, the FPAA-based controller helps the human user complete teleoperation tasks that require high controller gains such as when a large impedance needs to be displayed against the user's hand. Also, an experimental object stiffness discrimination study shows that large sampling periods, necessitating low control gains for maintaining stability, lead to unacceptable task performance by the user; however, the users show an improved ability to discriminate the various objects if the teleoperation controller is implemented using an FPAA.

Deficits in go/no-go task performance in male undergraduate high-risk alcohol users are driven by speeded responding to go stimuli

ABSTRACTBackground: Response disinhibition plays an important role in addictive behaviors. However, results of studies on the performance on response inhibition tasks of individuals evidencing potentially problematic levels of alcohol drinking are mixed. Objectives: We assessed conditions under which persons with a relatively high risk of alcohol dependence show inhibition deficits in such tasks and investigated the nature of those deficits. Methods: Fifty-eight male undergraduate students, 27 of which were high-risk drinkers according to the Alcohol Use Disorders Identification Test, performed a go/no-go inhibition task with differing percentages of no-go trials (50% vs. 25%), stimulus presentation times (600 vs. 200 ms), and types of go and no-go stimuli (alcohol related vs. -unrelated). Response inhibition was indexed by response time (RT) to go trials and response accuracy on go and no-go trials. Results: There were no differences between low- and high-risk drinkers on any of the three outcome measures under the 600-ms stimulus presentation condition. Under the 200-ms condition, the high-risk drinkers showed faster RTs to go stimuli, and more errors on both go- and no-go trials than the low-risk drinkers, irrespective of type and percentage of no-go stimuli. However, the accuracy differences between the two groups disappeared after controlling for the RT on go trials, suggesting a speed-accuracy trade-off. Conclusion: High-risk drinkers’ response inhibition deficits are not restricted to alcohol-related cues and are especially likely to occur under conditions prompting fast responding. These findings could be used to inform treatment, suggesting the promotion of strategies aimed at preventing high-risk alcohol users from making quick decisions.

Adolescent Cannabis Use: What is the Evidence for Functional Brain Alteration?

Cannabis use typically commences during adolescence, a period during which the brain undergoes profound remodeling in areas that are high in cannabinoid receptors and that mediate cognitive control and emotion regulation. It is therefore important to determine the impact of adolescent cannabis use on brain function. We investigate the impact of adolescent cannabis use on brain function by reviewing the functional magnetic resonance imaging studies in adolescent samples. We systematically reviewed the literature and identified 13 functional neuroimaging studies in adolescent cannabis users (aged 13 to 18 years) performing working memory, inhibition and reward processing tasks. The majority of the studies found altered brain function, but intact behavioural task performance in adolescent cannabis users versus controls. The most consistently reported differences were in the frontal-parietal network, which mediates cognitive control. Heavier use was associated with abnormal brain function in most samples. A minority of studies controlled for the influence of confounders that can also undermine brain function, such as tobacco and alcohol use, psychopathology symptoms, family history of psychiatric disorders and substance use. Emerging evidence shows abnormal frontal-parietal network activity in adolescent cannabis users, particularly in heavier users. Brain functional alterations may reflect a compensatory neural mechanism that enables normal behavioural performance. It remains unclear if cannabis exposure drives these alterations, as substance use and mental health confounders have not been systematically examined.

Design and evaluation of line symbolizations for origin–destination flow maps

We present the results of a user study comparing variants of commonly used line symbolizations for directed origin–destination flow maps. Our design and evaluation consisted of five line symbolizations that employ a combination of following visual variables: arrowheads, origin–destination coloring (color hue, and value), line shortening, line width, tapered edges (varying width from wide to narrow, and narrow to wide), and curvature asymmetry and strength. To guide our evaluation, we used a task-by-type typology and chose four representative tasks that are commonly used in flow map reading: identifying dominant direction of flows, flows with the highest magnitude (volume), spatial focusing of long flows toward a destination, and clusters of high net-exports (net-outflow). We systematically analyzed user responses and task performance which we measured by task completion time and accuracy. We designed a web-based flow mapping and testing framework and recruited the participants from Amazon Mechanical Turk. To demonstrate the application and user experiment, we used 16 commodity flow data sets in the United States from 2007 and systematically rotated the layouts to evaluate the effect of layout orientation. From this study, we can conclude that there is potential usefulness for all of the five symbolizations we tested; however, the influence of the design on performance and perception depends on the type of the task. Also, we found that data and layout orientation have significant effects on performance and perception of patterns in flow maps which we attribute to the change in visual saliency of node and flow patterns in relation to the way users scan the map. We recommend that the choice of line symbolization should be guided by a task taxonomy which end users are expected to perform. We discuss various design trade-offs and recommendations and potential future work for designing and evaluating line symbolizations for flow mapping.

Prediction of individual learning curves across information visualizations

Confident usage of information visualizations is thought to be influenced by cognitive aspects as well as amount of exposure and training. To support the development of individual competency in visualization processing, it is important to ascertain if we can track users' progress or difficulties they might have while working with a given visualization. In this paper, we extend previous work on predicting in real time a user's learning curve--a mathematical model that can represent a user's skill acquisition ability--when working with a visualization. First, we investigate whether results we previously obtained in predicting users' learning curves during visualization processing generalize to a different visualization. Second, we study to what extent we can make predictions on a user's learning curve without information on the visualization being used. Our models leverage various data sources, including a user's gaze behavior, pupil dilation, and cognitive abilities. We show that these models outperform a baseline that leverages knowledge on user task performance so far. Our best performing model achieves good accuracies in predicting users' learning curves even after observing users' performance on a few tasks only. These results represent an important step toward understanding how to support users in learning a new visualization.

A Multimodal Interaction Design Guidelines for VR Foot Reflexology Therapy Application

The proliferation of multimodal interaction research have presented numerous advantages which include enhancement of realism, increased efficiency in user task performance and the achievement of an interactivity that is relatively identical to the conventional human to human interactions. Haptic, visual, and aural interactions have been widely utilized and applied in various domains ranging from military and scientific visualization realms into more multidisciplinary areas, such as art and culture, education, archeology, and complementary medicine. In Virtual reality stress therapy application particularly in the proliferation towards the virtualization and simulation of the traditional foot reflexology therapy, little is understood on the haptic, visual and aural interactive nature of the therapy and to what extent they contribute in the patients' relaxation and stress relief, which hinders the system developers from obtaining the appropriate system design requirements for foot reflexology virtual stress therapy applications. This paper presents an exploratory study that examines from both the patients' and practitioners' perspective, the haptic, visual and aural interactive nature in foot reflexology domain since the practices promote relaxation and stress relief. The study explored 2 traditional foot reflexology sessions; audio recorded semi-structured interview was used to collect data from the participant's while the session was going on for analysis. The study findings presented the haptic, visual and aural interactive nature involved from the patients' and practitioners' perspective, and a higher level design requirements for the haptic, aural, visual and general interactivity extracted from the study findings as well as from literature. Implications for future research are also discussed.

Visualizing Sets with Linear Diagrams

This paper presents the first design principles that optimize the visualization of sets using linear diagrams. These principles are justified through empirical studies that evaluate the impact of graphical features on task performance. Linear diagrams represent sets using straight line segments, with line overlaps corresponding to set intersections. This study builds on recent empirical research, which establishes that linear diagrams can be superior to prominent set visualization techniques, namely Euler and Venn diagrams. We address the problem of how to best visualize overlapping sets using linear diagrams. To solve the problem, we investigate which graphical features of linear diagrams significantly impact user task performance. To this end, we conducted seven crowdsourced empirical studies involving a total of 1,760 participants. These studies allowed us to identify the following design principles, which significantly aid task performance: use a minimal number of line segments, use guidelines where overlaps start and end, and draw lines that are thin as opposed to thick bars. We also evaluated the following graphical properties that did not significantly impact task performance: color, orientation, and set order. The results are brought to life through a freely available software implementation that automatically draws linear diagrams with user-controlled graphical choices. An important consequence of our research is that users are now able to create effective visualizations of sets automatically, thus improving human--computer interaction.

Exploratory information searching in the enterprise: A study of user satisfaction and task performance

No prior research has been identified that investigates the causal factors for workplace exploratory search task performance. The impact of user, task, and environmental factors on user satisfaction and task performance was investigated through a mixed methods study with 26 experienced information professionals using enterprise search in an oil and gas enterprise. Some participants found 75% of high‐value items, others found none, with an average of 27%. No association was found between self‐reported search expertise and task performance, with a tendency for many participants to overestimate their search expertise. Successful searchers may have more accurate mental models of both search systems and the information space. Organizations may not have effective exploratory search task performance feedback loops, a lack of learning. This may be caused by management bias towards technology, not capability, a lack of systems thinking. Furthermore, organizations may not “know” they “don't know” their true level of search expertise, a lack of knowing. A metamodel is presented identifying the causal factors for workplace exploratory search task performance. Semistructured qualitative interviews with search staff from the defense, pharmaceutical, and aerospace sectors indicates the potential transferability of the finding that organizations may not know their search expertise levels.

Do human cognitive differences in information processing affect preference and performance of CAPTCHA?

Abstract A Completely Automated Public Turing test to tell Computers and Humans Apart (CAPTCHA) is a widely used security defense mechanism that is utilized by service providers to determine whether the entity interacting with their system is a human and not a malicious agent. Common design practices of current CAPTCHA schemes barely take into account cultural, contextual, and individual cognitive characteristics and abilities of users. Motivated by recent research which underpins the necessity for designing more user-friendly CAPTCHA, this paper investigates the effect of users’ cognitive styles and cognitive processing abilities towards preference and task performance of CAPTCHA challenges. In the frame of the reported research, two user studies were conducted. The first study (n=131) explored the effect of users’ cognitive styles (Verbal/Imager) on user preference and task performance of two complementary types of CAPTCHA mechanisms; text-recognition and image-recognition. The second study (n=125) explored the effect of users’ cognitive processing abilities (speed of processing, controlled attention, working memory capacity) on task performance in regards with different levels of complexity of both text-recognition and image-recognition CAPTCHA. Analysis of results revealed interaction effects of users’ cognitive processing characteristics towards preference and performance of CAPTCHA, suggesting that individual differences at such an intrinsic level are important to be considered for designing more usable and user-centric CAPTCHA challenges.

Effect of elastic touchscreen and input devices with different softness on user task performance and subjective satisfaction

This study explored the relationship between the softness of two output displays (glass and elastic surfaces) and three direct input devices (fingertip and two types of styli, with acrylic and rubber tips) in terms of user task performance and satisfaction in tapping tasks. In a within-subjects experiment, 44 participants were asked to play a touch-based game as quickly and accurately as possible across combinations of the two independent variables. After finishing tasks in each condition, they filled out an evaluation questionnaire regarding tactile satisfaction, which addressed perceived feeling of pressing down on the surface, pleasantness, helpfulness, finger fatigue, and degree of elasticity. The results of this experiment show a main effect of touchscreen type on task performance, whereas input device type shows main effects on both task performance and satisfaction. Measures of five key sub-factors of subjective satisfaction are explained, and the practical implications of these findings are discussed. The two main stages of pre-interaction and post-interaction are observed.The combination of soft touchscreen and fingers shows the fastest task performance.An outstanding task performance does not guarantee a high level of user satisfaction.The elastic-surfaced display with the rubber-tipped stylus is the best for tapping.The glass-surfaced display with the acrylic-tipped stylus is the worst combination.

Finding Waldo: Learning about Users from their Interactions.

Visual analytics is inherently a collaboration between human and computer. However, in current visual analytics systems, the computer has limited means of knowing about its users and their analysis processes. While existing research has shown that a user's interactions with a system reflect a large amount of the user's reasoning process, there has been limited advancement in developing automated, real-time techniques that mine interactions to learn about the user. In this paper, we demonstrate that we can accurately predict a user's task performance and infer some user personality traits by using machine learning techniques to analyze interaction data. Specifically, we conduct an experiment in which participants perform a visual search task, and apply well-known machine learning algorithms to three encodings of the users' interaction data. We achieve, depending on algorithm and encoding, between 62% and 83% accuracy at predicting whether each user will be fast or slow at completing the task. Beyond predicting performance, we demonstrate that using the same techniques, we can infer aspects of the user's personality factors, including locus of control, extraversion, and neuroticism. Further analyses show that strong results can be attained with limited observation time: in one case 95% of the final accuracy is gained after a quarter of the average task completion time. Overall, our findings show that interactions can provide information to the computer about its human collaborator, and establish a foundation for realizing mixed-initiative visual analytics systems.

품질 관점에서 가상 데스크탑 인프라(VDI)의 만족과 성과, 그리고 업무 상호의존성의 조절효과

With the explosive proliferation of smart devices that may be connected to the fast Internet, virtual desktop interfaces(VDI) are being implemented in enterprises allowing employees to work anywhere anytime in the same technological environment. As this kind of systems are intended to raise the work performance, a research model is constructed from the review of research literature on service quality and work design. The model consists of VDI system service quality (ubiquity, availability, compatibility, security and ease of use), system satisfaction, task performance and job satisfaction. As VDI is designed as a support system for cooperative work, the task interdependence adopted from the work design theory is postulated here as moderating the effect of user satisfaction on task performance and job satisfaction. 147 data points were collected by a survey of VDI users in a global firm and subjected to PLS analysis. Interestingly, ubiquity and compatibility seem to be not statistically significant for user satisfaction while availability, security and ease of use of VDI system are statistically significant. Task interdependence is found to be a relatively strong mediator between system user satisfaction and task performance, however, interestingly, the coefficient come out as negative. This may indicate that the influence of VDI user satisfaction on task performance would not be high in highly interdependent tasks. Considering that VDI is intended for supporting `interdependence` in cooperative work, this finding is a bit surprising. In-depth discussions are made in the discussion followed by future research directions.