Augmented Reality Framework to Measure and Analyze Eye–Hand Coordination in Stroke Patients with Unilateral Neglect: Proof-of-Concept Study

S161. Eye-hand coordination in hereditary spinocerebellar degeneration

Recently, Augmented Reality (AR) is an emerging technology in many mobile applications. Mobile AR was defined as a medium for displaying information merged with the real world environment mapped with augmented reality surrounding in a single view. There are four main types of mobile augmented reality interfaces and one of them are multimodal interfaces. Multimodal interface processes two or more combined user input modes (such as speech, pen, touch, manual gesture, gaze, and head and body movements) in a coordinated manner with multimedia system output. In multimodal interface, many frameworks have been proposed to guide the designer to develop a multimodal applications including in augmented reality environment but there has been little work reviewing the framework of adaptive multimodal interface in mobile augmented reality. The main goal of this study is to propose a conceptual framework to illustrate the adaptive multimodal interface in mobile augmented reality. We reviewed several frameworks that have been proposed in the field of multimodal interfaces, adaptive interface and augmented reality. We analyzed the components in the previous frameworks and measure which can be applied in mobile devices. Our framework can be used as a guide for designers and developer to develop a mobile AR application with an adaptive multimodal interfaces.Recently, Augmented Reality (AR) is an emerging technology in many mobile applications. Mobile AR was defined as a medium for displaying information merged with the real world environment mapped with augmented reality surrounding in a single view. There are four main types of mobile augmented reality interfaces and one of them are multimodal interfaces. Multimodal interface processes two or more combined user input modes (such as speech, pen, touch, manual gesture, gaze, and head and body movements) in a coordinated manner with multimedia system output. In multimodal interface, many frameworks have been proposed to guide the designer to develop a multimodal applications including in augmented reality environment but there has been little work reviewing the framework of adaptive multimodal interface in mobile augmented reality. The main goal of this study is to propose a conceptual framework to illustrate the adaptive multimodal interface in mobile augmented reality. We reviewed several frameworks that hav...

Walking trajectory and hand movements in unilateral left neglect: A vestibular hypothesis

Active and passive activation of left limbs: Influence on visual and sensory neglect

The purpose of the research is to compare the impact of students’ learning performance between an augmented reality (AR) environment and in-class environment. To create an efficient AR environment, we used Microsoft HoloLens, which is the next generation of a see-through holographic computer. We developed the AR learning module: ergonomic guidelines for manual material handling (MMH). We hypothesize that AR changes the way students understand the concepts of MMH. Our analysis includes a careful evaluation of student experimental skills during the learning activities. This new AR environment could allow students to engage hands-on training of MMH and strengthen their understanding. Student test score was used as metrics for performance assessment. We found a significant improvement on student understanding of MMH lecture after they used the AR module. The findings of this study indicated the potential benefits of using AR environment interfaces in engineering education and training.

Multimodal interaction in everyday life seems so effortless. However, a closer look reveals that such interaction is indeed complex and comprises multiple levels of coordination, from high-level linguistic exchanges to low-level couplings of momentary bodily movements both within an agent and across multiple interacting agents. A better understanding of how these multimodal behaviors are coordinated can provide insightful principles to guide the development of intelligent multimodal interfaces. In light of this, we propose and implement a research framework in which human participants interact with a virtual agent in a virtual environment. Our platform allows the virtual agent to keep track of the user's gaze and hand movements in real time, and adjust his own behaviors accordingly. An experiment is designed and conducted to investigate adaptive user behaviors in a human-agent joint attention task. Multimodal data streams are collected in the study including speech, eye gaze, hand and head movements from both the human user and the virtual agent, which are then analyzed to discover various behavioral patterns. Those patterns show that human participants are highly sensitive to momentary multimodal behaviors generated by the virtual agent and they rapidly adapt their behaviors accordingly. Our results suggest the importance of studying and understanding real-time adaptive behaviors in human-computer multimodal interactions.

As a vital skill in an evolving world, interception of moving objects relies on accurate prediction of target motion. In natural circumstances, active gaze shifts often accompany hand movements when exploring targets of interest, but how eye and hand movements are coordinated during manual interception and their dependence on visual prediction remain unclear. Here, we trained gaze-unrestrained monkeys to manually intercept targets appearing at random locations and circularly moving with random speeds. We found that well-trained animals were able to intercept the targets with adequate compensation for both sensory transmission and motor delays. Before interception, the animals' gaze followed the targets with adequate compensation for the sensory delay, but not for extra target displacement during the eye movements. Both hand and eye movements were modulated by target kinematics, and their reaction times were correlated. Moreover, retinal errors and reaching errors were correlated across different stages of reach execution. Our results reveal eye-hand coordination during manual interception, yet the eye and hand movements may show different levels of prediction based on the task context. NEW & NOTEWORTHY Here we studied the eye-hand coordination of monkeys during flexible manual interception of a moving target. Eye movements were untrained and not explicitly associated with reward. We found that the initial saccades toward the moving target adequately compensated for sensory transmission delays, but not for extra target displacement, whereas the reaching arm movements fully compensated for sensorimotor delays, suggesting that the mode of eye-hand coordination strongly depends on behavioral context.

Patients with unilateral neglect following right hemisphere damage may have difficulty in moving towards contralesional targets. To test the hypothesis that this impairment arises from competing motor programs triggered by irrelevant ipsilesional stimuli, we examined 16 right hemisphere patients, eight with left visual neglect and eight without, in addition to eight healthy control subjects. In experiment 1 subjects performed sequences of movements using their right hand to targets on the contralesional or ipsilesional side of the responding limb. The locations of successive targets in each sequence were either predictable or unpredictable. In separate blocks of trials, targets appeared either alone or with a simultaneous distractor located at the immediately preceding target location. Neglect patients were significantly slower to execute movements to contralesional targets, but only for unpredictable movements and in the presence of a concurrent ipsilesional distractor. In contrast, healthy controls and right hemisphere patients without neglect showed no directional asymmetries of movement execution. In experiment 2 subjects were required to interrupt a predictable, reciprocating sequence of leftward and rightward movements in order to move to an occasional, unpredictable target that occurred either in the direction opposite to that expected, or in the same direction but twice the extent. Neglect patients were significantly slower in reprogramming the direction and extent of movements towards contralesional versus ipsilesional targets, and they also made significantly more errors when executing such movements. Right hemisphere patients without neglect showed a similar bias in reprogramming direction (but not extent) for contralesional targets, whereas healthy controls showed no directional asymmetry in either condition. On the basis of these findings we propose that neglect involves a competitive bias in favour of motor programs for actions directed towards ipsilesional versus contralesional events. We suggest that programming errors and increased latencies for contralesional movements arise because the damaged right hemisphere can no longer effectively inhibit the release of inappropriate motor programs towards ipsilesional events.

In a cyclical Fitts' task, hand movements transition from continuous to discrete movements when the Index of Difficulty (ID) increases. Moreover, at high ID (small target), the eyes saccade to and subsequently fixate the targets at every movement, while at low ID (large target) intermittent monitoring is used. By hypothesis, the (periodic) gaze shifts are abandoned for movement times shorter than about 0.350s due to systemic constraints (i.e., a refractory period and intrinsic latency). If so, the transition in eye and hand movements is independent. To investigate these issues, the present study examined the effects of changing ID via the targets' width or distance as well as hysteresis in eye-hand coordination. To this aim, 14 participants performed a cyclical Fitts' task while their hand and eye movements were recorded simultaneously. The results show that the transition in eye-hand synchronization (at 2.87 bit; 0.25s) and in hand dynamics (at 4.85 bit; 0.81s) neither co-occurred nor correlated. Some small width vs. distance dissociations and hysteresis effects were found, but they disappeared when eye-hand synchronization was viewed as a function of movement time rather than ID. This confirms that a minimal between-saccade time is the limiting factor in eye-hand synchronization. Additionally, the timing between the start of the hand movement and the saccade appeared to be relatively constant (at 0.15s) and independent of movement time, implying a constant delay that should be implemented in a dynamical model of eye-hand coordination.

Designing virtual characters that are capable of conveying a sense of personality is important for generating realistic experiences, and thus a key goal in computer animation research. Though the influence of gesture and body motion on personality perception has been studied, little is known about which attributes of hand pose and motion convey particular personality traits. Using the “Big Five” model as a framework for evaluating personality traits, this work examines how variations in hand pose and motion impact the perception of a character's personality. As has been done with facial motion, we first study hand motion in isolation as a requirement for running controlled experiments that avoid the combinatorial explosion of multimodal communication (all combinations of facial expressions, arm movements, body movements, and hands) and allow us to understand the communicative content of hands. We determined a set of features likely to reflect personality, based on research in psychology and previous human motion perception work: shape, direction, amplitude, speed, and manipulation. Then we captured realistic hand motion varying these attributes and conducted three perceptual experiments to determine the contribution of these attributes to the character's personalities. Both hand poses and the amplitude of hand motion affected the perception of all five personality traits. Speed impacted all traits except openness. Direction impacted extraversion and openness. Manipulation was perceived as an indicator of introversion, disagreeableness, neuroticism, and less openness to experience. From these results, we generalize guidelines for designing detailed hand motion that can add to the expressiveness and personality of characters. We performed an evaluation study that combined hand motion with gesture and body motion. Even in the presence of body motion, hand motion still significantly impacted the perception of a character's personality and could even be the dominant factor in certain situations.

Unilateral spatial neglect (USN) is a disorder characterized by the failure to report, respond to, or orient toward the contralateral side of space to a brain lesion. Current assessment methods often fail to discover milder forms, cannot differentiate between unilateral spatial neglect subtypes and lack ecological validity. There is also a need for treatment methods that target subtypes. Immersive virtual reality (VR) systems in combination with eye-tracking (ET) have the potential to overcome these shortcomings, by providing more naturalistic environments and tasks, with sensitive and detailed measures. This systematic review examines the state of the art of research on these technologies as applied in the assessment and treatment of USN. As we found no studies that combined immersive VR and ET, we reviewed these approaches individually. The review of VR included seven articles, the ET review twelve. The reviews revealed promising results. (1) All included studies found significant group-level differences for several USN measures. In addition, several studies found asymmetric behavior in VR and ET tasks for patients who did not show signs of USN in conventional tests. Particularly promising features were multitasking in complex VR environments and detailed eye-movement analysis. (2) No VR and only a few ET studies attempted to differentiate USN subtypes, although the technologies appeared appropriate. One ET study grouped USN participants using individual heatmaps, and another differentiated between subtypes on drawing tasks. Regarding (3) ecological validity, although no studies tested the prognostic validity of their assessment methods, VR and ET studies utilized naturalistic tasks and stimuli reflecting everyday situations. Technological characteristics, such as the field of view and refresh rate of the head-mounted displays, could be improved, though, to improve ecological validity. We found (4) no studies that utilized VR or ET technologies for USN treatment up until the search date of the 26th of February 2020. In conclusion, VR-ET-based systems show great potential for USN assessment. VR-ET holds great promise for treatment, for example, by monitoring behavior and adapting and tailoring to the individual person’s needs and abilities. Future research should consider developing methods for individual subtypes and differential diagnostics to inform individual treatment programs.

Conventional ultrasound-guided vascular access procedures are challenging due to the need for anatomical understanding, precise needle manipulation, and hand-eye coordination. Recently, augmented reality (AR)-based guidance has emerged as an aid to improve procedural efficiency and potential outcomes. However, its application in pediatric vascular access has not been comprehensively evaluated. We developed an AR ultrasound application, HoloUS, using the Microsoft HoloLens 2 to display live ultrasound images directly in the proceduralist's field of view. We presented our evaluation of the effect of using the Microsoft HoloLens 2 for point-of-care ultrasound (POCUS)-guided vascular access in 30 pediatric patients. A custom software module was developed on a tablet capable of capturing the moving ultrasound image from any ultrasound machine's screen. The captured image was compressed and sent to the HoloLens 2 via a hotspot without needing Internet access. On the HoloLens 2, we developed a custom software module to receive, decompress, and display the live ultrasound image. Hand gesture and voice command features were implemented for the user to reposition, resize, and change the gain and the contrast of the image. We evaluated 30 (15 successful control and 12 successful interventional) cases completed in a single-center, prospective, randomized study. The mean overall rendering latency and the rendering frame rate of the HoloUS application were 139.30ms and 30 frames per second, respectively. The average procedure completion time was 17.3% shorter using AR guidance. The numbers of puncture attempts and needle redirections were similar between the two groups, and the number of head adjustments was minimal in the interventional group. We presented our evaluation of the results from the first study using the Microsoft HoloLens 2 that investigates AR-based POCUS-guided vascular access in pediatric patients. Our evaluation confirmed clinical feasibility and potential improvement in procedural efficiency.

A better understanding of the human user’s expectations and sensitivities to the real-time behavior generated by virtual agents can provide insightful empirical data and infer useful principles to guide the design of intelligent virtual agents. In light of this, we propose and implement a research framework to systematically study and evaluate different important aspects of multimodal real-time interactions between humans and virtual agents. Our platform allows the virtual agent to keep track of the user’s gaze and hand movements in real time, and adjust his own behaviors accordingly. Multimodal data streams are collected in human-avatar interactions including speech, eye gaze, hand and head movements from both the human user and the virtual agent, which are then used to discover fine-grained behavioral patterns in human-agent interactions. We present a pilot study based on the proposed framework as an example of the kinds of research questions that can be rigorously addressed and answered. This first study investigating human-agent joint attention reveals promising results about the role and functioning of joint attention in human-avatar interactions.Keywordsembodied agentmultimodal interactionvisualization

Prism adaptation theory in unilateral neglect: motor and perceptual components

Three-dimensional sound's effectiveness in virtual reality (VR) environments has been widely studied. However, due to the big differences between VR and augmented reality (AR) systems in registration, calibration, perceptual difference of immersiveness, navigation, and localization, it is important to develop new approaches to seamlessly register virtual 3-D sound in AR environments and conduct studies on 3-D sound's effectiveness in AR context. In this paper, we design two experimental AR environments to study the effectiveness of 3-D sound both quantitatively and qualitatively. Two different tracking methods are applied to retrieve the 3-D position of virtual sound sources in each experiment. We examine the impacts of 3-D sound on improving depth perception and shortening task completion time. We also investigate its impacts on immersive and realistic perception, different spatial objects identification, and subjective feeling of human presence and collaboration. Our studies show that applying 3-D sound is an effective way to complement visual AR environments. It helps depth perception and task performance, and facilitates collaborations between users. Moreover, it enables a more realistic environment and more immersive feeling of being inside the AR environment by both visual and auditory means. In order to make full use of the intensity cues provided by 3-D sound, a process to scale the intensity difference of 3-D sound at different depths is designed to cater small AR environments. The user study results show that the scaled 3-D sound significantly increases the accuracy of depth judgments and shortens the searching task completion time. This method provides a necessary foundation for implementing 3-D sound in small AR environments. Our user study results also show that this process does not degrade the intuitiveness and realism of an augmented audio reality environment