Combined Eye Tracking Research Articles

Mitigating Unwarranted Forecasting Behavior in the Newsvendor Task

We use a combination of eye tracking and retrospective verbal protocol analysis (VPA) to map the decision making process in the newsvendor task. In Study 1, we found that decision makers in the classic repeated newsvendor task setting show a propensity to forecast demand using perceived trends in the past demand when actually forecasting is normatively inappropriate in this context. We explored two interventions to rectify this pattern of behavior. First, by suppressing the temporal dimension (Supp_Temp_Dim), that is, asking the participants to make newsvendor decisions for 20 different products for the same period instead of making the newsvendor decision 20 times for the same product, we observed a decrease in the extent of forecasting behavior. Second, by imposing standing orders (Stdg_Ord), we observed that it increased the forecasting horizon. In Study 2, we tested an intervention that sensitized the subjects to the critical fractile logic (Crit_Frac) and found that it improved financial performance. The forecasting behavior persisted but the orders now fluctuated around a higher mean than before. The improved understanding of the fundamental cognitive processes that lurk behind the observed decisions can translate into improved real world decision making.

Visuospatial exploration and art therapy intervention in patients with Parkinson’s disease: an exploratory therapeutic protocol

Though abnormalities of visuospatial function occur in Parkinson’s disease, the impact of such deficits on functional independence and psychological wellbeing has been historically under- recognized, and effective treatments for this impairment are unknown. These symptoms can be encountered at any stage of the disease, affecting many activities of daily living, and negatively influencing mood, self-efficacy, independence, and overall quality of life. Furthermore, visuospatial dysfunction has been recently linked to gait impairment and falls, symptoms that are known to be poor prognostic factors. Here, we aim to present an original modality of neurorehabilitation designed to address visuospatial dysfunction and related symptoms in Parkinson’s disease, known as “Art Therapy”. Art creation relies on sophisticated neurologic mechanisms including shape recognition, motion perception, sensory-motor integration, abstraction, and eye-hand coordination. Furthermore, art therapy may enable subjects with disability to understand their emotions and express them through artistic creation and creative thinking, thus promoting self-awareness, relaxation, confidence and self-efficacy. The potential impact of this intervention on visuospatial dysfunction will be assessed by means of combined clinical, behavioral, gait kinematic, neuroimaging and eye tracking analyses. Potential favorable outcomes may drive further trials validating this novel paradigm of neurorehabilitation.

Predictive saccades in children and adults: A combined fMRI and eye tracking study.

Saccades were assessed in 21 adults (age 24 years, SD = 4) and 15 children (age 11 years, SD = 1), using combined functional magnetic resonance imaging (fMRI) and eye-tracking. Subjects visually tracked a point on a horizontal line in four conditions: time and position predictable task (PRED), position predictable (pPRED), time predictable (tPRED) and visually guided saccades (SAC). Both groups in the PRED but not in pPRED, tPRED and SAC produced predictive saccades with latency below 80 ms. In task versus group comparisons, children’s showed less efficient learning compared to adults for predictive saccades (adults = 48%, children = 34%, p = 0.05). In adults brain activation was found in the frontal and occipital regions in the PRED, in the intraparietal sulcus in pPRED and in the frontal eye field, posterior intraparietal sulcus and medial regions in the tPRED task. Group–task interaction was found in the supplementary eye field and visual cortex in the PRED task, and the frontal cortex including the right frontal eye field and left frontal pole, in the pPRED condition. These results indicate that, the basic visuomotor circuitry is present in both adults and children, but fine-tuning of the activation according to the task temporal and spatial demand mature late in child development.

A Combined Eye-tracking and EEG Analysis on Chess Moves

Nowadays it is possible to disclose the human cognitive process with increasing details on a number of challenging mental tasks. This paper investigates the focus of visual attention and brain activation on chess moves between players with different proficiencies using a combined eye-tracking and electroencephalographic analysis. Experiments were carried out involving 51 questions related to the chess game, separated into 5 categories and with the participation of 32 volunteers, including professional (with ELO rating) and non-professional chess players. All volunteers' performance were assessed using a quantitative metric based on the answers accuracy and average response time. Our results clearly show that such combined approach allows the interpretation of significant statistical differences between the most proficient and less proficient groups, which would not be possible using the information from a single data source only.

Frontotemporal stimulation modulates semantically-guided visual search during confrontation naming: A combined tDCS and eye tracking investigation

Transcranial direct current stimulation (tDCS) was paired with eye tracking to elucidate contributions of frontal, temporoparietal and anterior temporal cortex to early visual search patterns during picture naming (e.g., rapid visual scanning to diagnostic semantic features). Neurotypical adults named line drawings of objects prior to and following tDCS in three separate sessions, each employing a unique electrode montage. The gaze data revealed montage by stimulation (pre/post) interaction effects characterized by longer initial visual fixations (mean difference = 89 ms; Cohen’s d = .8) and cumulative fixation durations (mean difference = 98 ms; Cohen’s d = .9) on key semantic features (e.g., the head of an animal) after cathodal frontotemporal stimulation relative to the pre-stimulation baseline. We interpret these findings as reflecting a tDCS-induced modulation of semantic contributions of the anterior temporal lobe(s) to top-down influences on object recognition. Further, we discuss implications for the optimization of tDCS for the treatment of anomia in aphasia.

Measuring the influence of map label density on perceived complexity: a user study using eye tracking

ABSTRACTWe combine eye tracking and a questionnaire-based approach to explore the influence of label density on the perceived visual complexity of maps. We design two experiments in which participants are asked to search for the names of point features on maps and to rate the map complexity and legibility for different label densities. Specifically, we conduct a highly controlled experiment in which all the map variables except the label density are held constant (the controlled experiment). Then, we conduct a second experiment following the same protocol but using real maps as visual stimuli (the real-map experiment) to verify if the results of the controlled experiment were applicable to real maps. The results of both experiments indicate a significantly positive correlation between perceived visual complexity and label density and between the response time in visual search tasks and label density. Surprisingly, we observe a significant inverse correlation between the label density and two eye movement parameters (fixation duration and fixation frequency) between the two experiments. We discuss how the variables of real maps might have affected these eye movement parameters and why the results of the two experiments are inconsistent. Our findings suggest that eye tracking parameters are not reliable indicators of map complexity. These empirical results can be helpful to future map design and map complexity investigation.

Consumer preferences for organic labels in Germany using the example of apples – Combining choice-based conjoint analysis and eye-tracking measurements

Abstract Organic food products are considered as credence goods. Hence, organic labels are important visual stimuli for consumers to determine how a good was produced. Therefore, this study aims to contribute to the literature by examining the visual attention patterns and preferences of consumers in a choice experiment by measuring their eye movements using eye-tracking technology and comparing the results with the findings from choice-based conjoint analysis (CBCA) and other stated preference methods. For this reason, 73 consumers participated in an experiment in 2017 in Germany. The combination of eye-tracking and CBCA can help to overcome some major limitations of choice experiments. For example, low levels of a attribute's part-worth utility can be caused by the participants low preference regarding that attribute or that he/she just haven't viewed the attribute. Our findings indicate that consumers who gain greater utilities from specific attributes of a good also attend to them visually to a greater extent. The study's results also suggest that a combination of stated preference methods with eye-tracking technology has the potential of solving some major limitations of these methods such as social desirability, memory limitations and lack of visual attention to specific product characteristics.

How to optimize sales flyers – novel experiment design

PurposeThe purpose of this paper is to show how analysing sales flyers with a combination of eye tracking, measurement of emotions, interview and content analysis can give an in-depth understanding on how different design aspects influence sales flyers’ effectiveness as a communication tool. The paper shows the relationship between different sales flyer design principles and a person’s preference towards it, as well as the intent to read it.Design/methodology/approachThe paper chose for pilot study using eye tracking and emotions measurement to analyse retail sales flyers. In addition, interviews and content analysis were conducted to fully understand which aspects of sales flyer design influenced consumers.FindingsThe paper’s main findings are that sales flyers that evoke more positive emotions are prone to be chosen, and the attention and the view time of content pages is related to the number of elements on the page, page coherence and the location of the offers.Research limitations/implicationsThis research uses eye tracking were sales flyers are shown on screen, which is not a natural way to read sales flyers. Future research should aim to test this methodology and prepositions in the natural environment.Practical implicationsThe paper includes implications for designing better sales flyers.Originality/valueTo the authors’ knowledge, sales flyers have never been studied with a research design combining eye tracking, measurement of emotions, interview, content analysis and preferences.

Food specific inhibitory control under negative mood in binge-eating disorder: Evidence from a multimethod approach.

Inhibitory control has been discussed as a developmental and maintenance factor in binge-eating disorder (BED). The current study is the first aimed at investigating inhibitory control in a negative mood condition on a psychophysiological and behavioral level in BED with a combination of electroencephalography (EEG) and eye tracking (ET). We conducted a combined EEG and ET study with overweight individuals with BED (BED+, n=24, mean age=31, mean BMI=35 kg/m2 ) and without BED (BED-, n=23, mean age=28, mean BMI=35 kg/m2 ) and a normal-weight (NWC, n=26, mean age 28, mean BMI=22 kg/m2 ) control group. We assessed self-report data regarding impulsivity and emotion regulation as well as the processing of food stimuli under negative mood in an antisaccade task. Main outcome variables comprise event-related potentials (ERP) regarding conflict processing (N2) and performance monitoring (error-related negativity [ERN/Ne]) assessed by EEG and inhibitory control (errors in the first and second saccade) assessed by ET. BED+ patients reported increased impulsivity and higher emotion regulation difficulties compared with the other groups. The eye tracking data revealed impaired inhibitory control in BED+ compared with both control groups. Further, we found preliminary evidence from EEG recordings that conflict processing might be less thorough in the BED+ sample as well as in the NWC sample. In the BED+ sample this might be connected to the inhibitory control deficits on behavioral level. While the BED- sample showed increased conflict processing latencies (N2 latencies), which might indicate a compensation mechanism, the BED+ sample did not show such a mechanism. Performance monitoring (ERN/Ne latencies and amplitudes) was not impaired in the BED+ sample compared with both control samples. Participants with BED reported higher impulsivity and lower emotion regulation capacities. The combined investigation of electrocortical processes and behavior contributes to an advanced understanding of behavioral and electrocortical processes underlying inhibitory control in BED. Inhibitory control and negative mood, probably amplified by emotion regulation deficits, should be addressed further in the investigation and treatment of BED.

Automatic Classification of Users' Health Information Need Context: Logistic Regression Analysis of Mouse-Click and Eye-Tracker Data.

BackgroundUsers searching for health information on the Internet may be searching for their own health issue, searching for someone else’s health issue, or browsing with no particular health issue in mind. Previous research has found that these three categories of users focus on different types of health information. However, most health information websites provide static content for all users. If the three types of user health information need contexts can be identified by the Web application, the search results or information offered to the user can be customized to increase its relevance or usefulness to the user.ObjectiveThe aim of this study was to investigate the possibility of identifying the three user health information contexts (searching for self, searching for others, or browsing with no particular health issue in mind) using just hyperlink clicking behavior; using eye-tracking information; and using a combination of eye-tracking, demographic, and urgency information. Predictive models are developed using multinomial logistic regression.MethodsA total of 74 participants (39 females and 35 males) who were mainly staff and students of a university were asked to browse a health discussion forum, Healthboards.com. An eye tracker recorded their examining (eye fixation) and skimming (quick eye movement) behaviors on 2 types of screens: summary result screen displaying a list of post headers, and detailed post screen. The following three types of predictive models were developed using logistic regression analysis: model 1 used only the time spent in scanning the summary result screen and reading the detailed post screen, which can be determined from the user’s mouse clicks; model 2 used the examining and skimming durations on each screen, recorded by an eye tracker; and model 3 added user demographic and urgency information to model 2.ResultsAn analysis of variance (ANOVA) analysis found that users’ browsing durations were significantly different for the three health information contexts (P<.001). The logistic regression model 3 was able to predict the user’s type of health information context with a 10-fold cross validation mean accuracy of 84% (62/74), followed by model 2 at 73% (54/74) and model 1 at 71% (52/78). In addition, correlation analysis found that particular browsing durations were highly correlated with users’ age, education level, and the urgency of their information need.ConclusionsA user’s type of health information need context (ie, searching for self, for others, or with no health issue in mind) can be identified with reasonable accuracy using just user mouse clicks that can easily be detected by Web applications. Higher accuracy can be obtained using Google glass or future computing devices with eye tracking function.

Using Eye Tracking to Detect the Effects of Clutter on Visual Search in Real Time

Display clutter causes decrements in visual search performance and can be a threat to safety and efficiency in complex, data-rich domains. Addressing the problem requires a means to detect the presence of clutter in real time, predict its effects, and then trigger countermeasures before breakdowns in information search can occur. Eye tracking is a promising technique for achieving these goals; however, to date, it has been used almost exclusively offline for display evaluation purposes. The goal of this research was instead to develop and evaluate models that combine eye tracking metrics to detect the effects of clutter early on in the search process. Participants were asked to locate targets in a simulated graphics program. Three eye tracking metrics—scanpath length, mean saccade amplitude, and mean fixation duration—were calculated over a 3-second time window. These metrics were then used as input to a set of logistic regression models to predict whether users’ response time will be relatively long or short. The accuracy of the models averaged 75% and the true positive rate was above 90%, with an ability to predict response time as early as 3.6 s into the visual search task. The results of this study confirm that eye tracking metrics can be used to predict the effects of display clutter in real time. They add to the knowledge base in attention and eye tracking, and they ultimately contribute to the design of adaptive displays that lead to improved operator performance.

The order of information processing alters economic gain-loss framing effects

Adaptive decision making requires analysis of available information during the process of choice. In many decisions that information is presented visually – which means that variations in visual properties (e.g., salience, complexity) can potentially influence the process of choice. In the current study, we demonstrate that variation in the left-right positioning of risky and safe decision options can influence the canonical gain-loss framing effect. Two experiments were conducted using an economic framing task in which participants chose between gambles and certain outcomes. The first experiment demonstrated that the magnitude of the gain-loss framing effect was greater when the certain option signaling the current frame was presented on the left side of the visual display. Eye-tracking data during task performance showed a left-gaze bias for initial fixations, suggesting that the option presented on the left side was processed first. Combination of eye-tracking and choice data revealed that there was a significant effect of direction of first gaze (i.e. left vs. right) as well as an interaction between gaze direction and identity of the first fixated information (i.e. certain vs. gamble) regardless of frame. A second experiment presented the gamble and certain options in a random order, with a temporal delay between their presentations. We found that the magnitude of gain-loss framing was larger when the certain option was presented first, regardless of left and right positioning, only in individuals with lower risk-taking tendencies. The effect of presentation order on framing was not present in high risk-takers. These results suggest that the sequence of visual information processing as well as their left-right positioning can bias choices by changing the impact of the presented information during risky decision making.

Visual Attention to Eco-Labels Predicts Consumer Preferences for Pollinator Friendly Plants

Recent declines in pollinator populations have led to widespread concern due to their impact on food/crop production and the environment. Contrary to growing interest in the use of insecticides in urban landscapes, the relationship between pollinator-related eco-labels, visual attention, and preferences for plants is less understood. The present study combines eye tracking and stated preference experiments to examine the effects of pollinator-related labels on consumers’ preferences and willingness to pay. Results show that the pollinator-friendly attribute positively correlates with consumers’ purchasing decisions and visual attention supports that relationship. Implications of mandatory labeling of pesticide content for the horticultural industry are discussed.

P 29 Brain mechanisms for phenomenal causation and oddball-perceptions – combining fMRI and high-frequent eye-tracking – A potential fundament for the first perceptual treatment of psychotic symptoms?

Brain mechanisms for phenomenal causation and oddball-perceptions – combining fMRI and high-frequent eye-tracking: a potential fundament for the first perceptual treatment of psychotic symptoms? Causality is a fundamental aspect of human perception, enabling us to make sense of events occurring in the world. Causality is also an often underestimated, although critical, feature of agency attribution, a fundamental cognitive function that is well-known to be perturbed in psychosis. The neuronal and psychophysiological correlates of causal perception are thus of particular interest for clinical neuroscientific research focused on psychotic symptomatology: Especially for research aimed at the development of potential cognitive or perceptual interventions to the core symptomatology of schizophrenia and malfunctionous agency attributions. The most important paradigm to investigate perceptual causality has been designed more than 70 years ago by Gestalt-Psychologist Albert Michotte (Michotte, 1946/1963). He has first shown that collisions of abstract moving stimuli (so-called “launch-events”), for example a collision of two billiard balls, evoke in healthy humans “Gestalt-perceptions of physical causality”, given that the motion event as a whole is contingent with the laws of mechanics/transfer of momentum (see for review e.g. Wagemans et al., 2006). Previous behavioral research indicates, that patients with psychosis show altered attributions of causality to launch-events (Tschacher and Kupper, 2006). Neuroimaging findings are further suggesting that the dysfunctional causality attributions occur in both physical and social event contexts, and are associated at the neural level with altered functional activity in the frontal cortex, as well as a disconnection of frontal and posterior cortex regions (Blakemore et al., 2003; Wende et al., 2014). However, an important confound of these studies is their reliance on explicit judgments or ratings of causality, making it impossible to infer from the results, if a higher-order cognitive (“top down”) or sensory-perceptual (“bottom up”) dysfunction is underlying the behavioral alterations. Here, eye-tracking can help to investigate the implicit component: Our paradigm is based on eye-movement monitoring at a real-time recording frequency (1 kHz, EyeLink 1000) during the observation of simultaneous launching displays (one launcher collides with two targets; one moves in a causal but non-colinear, the other one in a non-causal but colinear direction; Wende et al., 2015). Ocular (saccadic and pursuit) reactions to causal and non-causal target motions are thus for the first time compared within-trial, enabling us in a behavioral study, to confirm the Gestalt-perception hypothesis of causation (Michotte, 1946/1963), that the human visual system has encapsulated a rapidly operating and sub-attentional mechanism, which accomplishes an implicit detection of physical causality, as distinct from non-causality (“oddballs”): Both saccades (latencies) and smooth-pursuit (velocity) were biased towards the causal target motion (Wende et al., 2015). As a next step, we now aim to combine eye tracking and fMRI in order to investigate the functional neuroanatomy of this causality perception module in both healthy subjects and patients with psychosis. The long-term aim of our study set is the development of an interactive launching paradigm, to train causal inference at the sensory level in psychotic patients.

The first- and second-language age of acquisition effect in first- and second-language book reading

Abstract The age of acquisition (AoA) effect in first/monolingual language processing has received much attention in psycholinguistic research. However, AoA effects in second language processing were only investigated rarely. In the current study, we investigated first (L1) and second language (L2) AoA effects in a combined eye tracking and mega study approach. We analyzed data of a corpus of eye movements to assess the time course of AoA effects on bilingual reading. We found an effect of L2 AoA in both early and late measures of L2 reading: fixation times were faster for words that were learned earlier in L2. This suggests that the L2 AoA effect has an influence throughout the entire L2 reading process, analogous to the L1 AoA effect. However, we are also the first to find an early effect of L1 AoA on L2 processing: if the L1 translation of the L2 word was learned earlier, the L2 word was also read faster. We discuss the implications of these findings for two important hypotheses that offer an explanation for the AoA effect: the mapping and semantic hypothesis. We propose that the current results suggest an integration between these accounts.

Isolating Discriminant Neural Activity in the Presence of Eye Movements and Concurrent Task Demands.

A growing number of studies use the combination of eye-tracking and electroencephalographic (EEG) measures to explore the neural processes that underlie visual perception. In these studies, fixation-related potentials (FRPs) are commonly used to quantify early and late stages of visual processing that follow the onset of each fixation. However, FRPs reflect a mixture of bottom-up (sensory-driven) and top-down (goal-directed) processes, in addition to eye movement artifacts and unrelated neural activity. At present there is little consensus on how to separate this evoked response into its constituent elements. In this study we sought to isolate the neural sources of target detection in the presence of eye movements and over a range of concurrent task demands. Here, participants were asked to identify visual targets (Ts) amongst a grid of distractor stimuli (Ls), while simultaneously performing an auditory N-back task. To identify the discriminant activity, we used independent components analysis (ICA) for the separation of EEG into neural and non-neural sources. We then further separated the neural sources, using a modified measure-projection approach, into six regions of interest (ROIs): occipital, fusiform, temporal, parietal, cingulate, and frontal cortices. Using activity from these ROIs, we identified target from non-target fixations in all participants at a level similar to other state-of-the-art classification techniques. Importantly, we isolated the time course and spectral features of this discriminant activity in each ROI. In addition, we were able to quantify the effect of cognitive load on both fixation-locked potential and classification performance across regions. Together, our results show the utility of a measure-projection approach for separating task-relevant neural activity into meaningful ROIs within more complex contexts that include eye movements.

Contextual cueing improves attentional guidance, even when guidance is supposedly optimal.

Visual search through previously encountered contexts typically produces reduced reaction times compared with search through novel contexts. This contextual cueing benefit is well established, but there is debate regarding its underlying mechanisms. Eye-tracking studies have consistently shown reduced number of fixations with repetition, supporting improvements in attentional guidance as the source of contextual cueing. However, contextual cueing benefits have been shown in conditions in which attentional guidance should already be optimal-namely, when attention is captured to the target location by an abrupt onset, or under pop-out conditions. These results have been used to argue for a response-related account of contextual cueing. Here, we combine eye tracking with response time to examine the mechanisms behind contextual cueing in spatially cued and pop-out conditions. Three experiments find consistent response time benefits with repetition, which appear to be driven almost entirely by a reduction in number of fixations, supporting improved attentional guidance as the mechanism behind contextual cueing. No differences were observed in the time between fixating the target and responding-our proxy for response related processes. Furthermore, the correlation between contextual cueing magnitude and the reduction in number of fixations on repeated contexts approaches 1. These results argue strongly that attentional guidance is facilitated by familiar search contexts, even when guidance is near-optimal. (PsycINFO Database Record

The Effect of Organic Food Labels on Consumer Attention

ABSTRACTThis paper aims to reveal consumer perceptions of organic food with a special emphasis on the labelling of organic products. The objectives were to find out, whether or not a significant difference between men´s and women´s perception existed. Two groups (88 subjects in the experimental group, 59 subjects in the control group) were shown various food labels; with organic product labelling (experimental) and without (control). Results suggest the importance of organic labelling using eye-tracking data and additional in-depth interviews to discover the importance of the labelling for participants. Based on eye-tracking data, the greatest visual attention was paid to product brand, graphic elements and organic labelling claims with additional information, such as a quantity or flavour, also being important. Additional in-depth interviews discovered that labelling, particularly of the origin, is important for most consumers. Contrary to what was expected, eye-tracking showed very little difference in the perception of organic product labelling based on gender. Although organic food is being purchased by more people, 50% of participants mentioned, during interviews, that they buy organic food only occasionally. The combination of eye-tracking and in-depth interviews shows that organic product labelling can play a role in decision-making, but regardless 27% of participants do not care about the organic quality labels or don’t pay attention to them.

Eyes on the frame

PurposeUsing the internet parallel to or after television (TV) consumption changes the way people receive news. The way information is framed by the media has been found to influence the behavior of news recipients. The purpose of this paper is to hypothesize that the exposure to TV media frames would affect a lay audience’s online information-seeking behavior.Design/methodology/approachIn an experiment combining eye tracking and content analysis, participants (n=72) were exposed to one of three TV clips with different media frames (based on a full-sample content analysis) that focused on Alzheimer’s disease. After exposure, participants informed themselves about the issue online. Eye tracking allows to investigate whether individuals mainly scan information, or whether they compute information on a higher level of attention (use more thorough deliberate comparison of information and really reading information).FindingsThree different frames of online content were identified. Framing was found to influence the individual online searching and reading of information on a descriptive level (entering search words and viewing website content) to some degree, but not on a procedural level (such as selecting online search results).Research limitations/implicationsThis study makes a significant contribution to the literature embedding an established theoretical process like framing effects into the internet literature. Regarding the broader theoretical context, this study shed some light on cross-media framing effects on online behavior. Applying the psychological perspective of framing theory to explain and predict online searching behavior is beneficial for specific types of online search behavior. Main limitations are the not representative student sample and the forced task that participants had to inform themselves about Alzheimer’s disease online.Practical implicationsThe results have practical implications for the creation of TV-related websites. There can be a positive, profitable synergy of TV and online websites. The websites can complement the TV programs with the focus on information needs of the recipients depending on the TV activated audience frames. Therefore, media managers would do well to plan the contents of their websites as internet-based resources that meet the activated information needs.Originality/valueThis study is among the first to investigate the framing effects of TV on the online information searching behavior of individuals. A deeper understanding of how media frames, especially from TV, are affecting online information seeking will allow researchers to better explain and predict online user behavior and information needs. But still, more research is needed.

Multi-modal user interface combining eye tracking and hand gesture recognition

Many studies on eye tracking have been conducted in diverse research areas. Nevertheless, eye tracking continues to be limited by low accuracy and a severe vibration problem due to pupil tremors. Furthermore, because almost all selection interactions, such as click events, use a dwell-time or eye-blinking method, eye tracking presents issues for both time consumption and involuntary blinking. In this paper, we therefore propose a multi-modal interaction method using a combination of eye tracking and hand gesture recognition with the commercial hand gesture controller. This method performs global and intuitive navigation using eye tracking, and local and detailed navigation using hand gesture controller. It supports intuitive hand gestures for mouse-button clicking. Experimental results indicate that the targeting time for small points is significantly improved using the proposed method. Especially, the proposed method has advantages in large display with high spatial resolution environment. Also, the proposed clicking interaction and modality switching concept showed accurate recognition rate and positive training effect, respectively.