Multimodal Eye Research Articles

Media construction of pseudo-events: Lula da Silva at the Portugal’s parliament during a historic annual celebration

The Carnation Revolution (25th April 1974) has changed the course of Portugal, leading it to transition from an autocratic to a democratic Government. Every year, celebrations are conducted at the national Parliament. In 2023, it counted with another guest: Lula da Silva, Brazil’s ongoing president. Considering the notion of cultural memory and media construction of social reality through media events, this article notes Lula da Silva’s presence and exceptional address was portrayed on newspaper front pages of several journalistic media outlets. Therefore, it aims at understanding meaning and discourse production with Lula da Silva’s visit to Portugal on the 25th April 2023 through media coverage and first page highlight. Lula da Silva is noticed by the front pages under analysis as a controversial, entertaining, and eventful celebrity linked to specific positioned actors that is invited to the Parliament and prompts indirectly and directly discontentment and, at the same time, support. Also, it discusses how such spotlight over Lula da Silva turned that visit and Parliament’s address it into a pseudo-event. Adopting a Multimodal Critical Discourse Analysis approach, this research work intends to offer a multimodal eye upon those front pages and critically reflect on populism, coloniality, cultural memory, and journalism.

ShouldAR: Detecting Shoulder Surfing Attacks Using Multimodal Eye Tracking and Augmented Reality

Shoulder surfing attacks (SSAs) are a type of observation attack designed to illicitly gather sensitive data from "over the shoulder" of victims. This attack can be directed at mobile devices, desktop screens, Personal Identification Number (PIN) pads at an Automated Teller Machine (ATM), or written text. Existing solutions are generally focused on authentication techniques (e.g., logins) and are limited to specific attack scenarios (e.g., mobile devices or PIN Pads). We present ShotjldAR, a mobile and usable system to detect SSAs using multimodal eye gaze information (i.e., from both the potential attacker and victim). ShouldAR uses an augmented reality headset as a platform to incorporate user eye gaze tracking, rear-facing image collection and eye gaze analysis, and user notification of potential attacks. In a 24-participant study, we show that the prototype is capable of detecting 87.28% of SSAs against both physical and digital targets, a two-fold improvement on the baseline solution using a rear-facing mirror, a widely used solution to the SSA problem. The ShouldAR approach provides an AR-based, active SSA defense that applies to both digital and physical information entry in sensitive environments.

Multimodal Eye Imaging, Retina Characteristics, and Psychological Assessment Dataset

The eyes provide insights into psychology, potentially offering a distinctive perspective for psychological health profiles. However, there exist a notable deficiency in datasets that simultaneously encompass eye features and psychological assessments. To address this gap, our study presents a dataset that included Fundus Photography, Psychological Assessment, Retina Characteristics, and Multimodal Imaging (FPRM). FPRM dataset comprise fundus images at different wavelengths (548 nm and 605 nm), image of oxygen saturation for the retina and 8 specific retinal vessels, videos of retinal blood flow and pupillary light reflex, along with 61 items of multimodal quantitative measurement from 384 participants. Additionally, it features psychological assessments across five dimensions (geriatric depression, generalized anxiety disorder, insomnia, activities of daily living, and deterioration), accompanied by fundus photographs and 6 items of retina characteristics from 1683 participants. FPRM dataset is the first to integrate multimodal ophthalmic data and psychological assessments, not only advancing the development of machine learning applications but also facilitating in-depth research into the relationship between eye health and psychological health profiles.

Oral Hypoxia-Inducible Factor 2α Inhibitor Belzutifan in Ocular von Hippel-Lindau Disease: Subgroup Analysis of the Single-Arm Phase 2 LITESPARK-004 Study

ObjectiveTo report the efficacy of oral HIF-2α inhibitor belzutifan in participants with von Hippel-Lindau disease-associated retinal hemangioblastomas in LITESPARK-004. DesignSubgroup analysis of the phase 2, single-arm, open-label LITESPARK-004 study. ParticipantsAdults with ≥1 von Hippel-Lindau disease-associated measurable renal cell carcinoma tumor not requiring immediate surgical intervention were eligible. Methods and InterventionParticipants received oral belzutifan 120 mg once daily until disease progression or unacceptable treatment-related toxicity. Main Outcome MeasuresEfficacy of belzutifan in retinal hemangioblastomas was a secondary end point, measured as response (improved, stable, or progressed) by independent reading center certified graders based on color fundus imaging performed every 12 weeks using the investigator’s preferred imaging standards. Additional assessments, where available, included optical coherence tomography and ultra-widefield fluorescein angiography. ResultsAmong 61 participants in LITESPARK-004, 12 had ≥1 evaluable active retinal hemangioblastoma in 16 eyes at baseline per independent reading center. As of April 1, 2022, the median follow-up for participants with ocular von Hippel-Lindau disease at baseline was 37.3 months. All 16 eyes were graded as improved, with a response rate of 100.0% (95% confidence intervals, 79.4–100.0). No new retinal hemangioblastomas or ocular disease progression were reported as of data cutoff date. Eight participants had additional multimodal eye assessments performed at the National Institutes of Health study site. Among this subgroup, 10 of 24 hemangioblastomas in 8 eyes of 6 participants measured ≥500 μm in greatest linear dimension at baseline and were further analyzed. All 10 hemangioblastomas had a mean area reduction of ≥15% by month 12 and ≥30% by month 24. ConclusionsBelzutifan showed promising activity against ocular von Hippel-Lindau disease, including capacity to control retinal hemangioblastomas, with effects sustained for >2 years while on treatment.

Confidence-aware multi-modality learning for eye disease screening

Multi-modal ophthalmic image classification plays a key role in diagnosing eye diseases, as it integrates information from different sources to complement their respective performances. However, recent improvements have mainly focused on accuracy, often neglecting the importance of confidence and robustness in predictions for diverse modalities. In this study, we propose a novel multi-modality evidential fusion pipeline for eye disease screening. It provides a measure of confidence for each modality and elegantly integrates the multi-modality information using a multi-distribution fusion perspective. Specifically, our method first utilizes normal inverse gamma prior distributions over pre-trained models to learn both aleatoric and epistemic uncertainty for uni-modality. Then, the normal inverse gamma distribution is analyzed as the Student’s t distribution. Furthermore, within a confidence-aware fusion framework, we propose a mixture of Student’s t distributions to effectively integrate different modalities, imparting the model with heavy-tailed properties and enhancing its robustness and reliability. More importantly, the confidence-aware multi-modality ranking regularization term induces the model to more reasonably rank the noisy single-modal and fused-modal confidence, leading to improved reliability and accuracy. Experimental results on both public and internal datasets demonstrate that our model excels in robustness, particularly in challenging scenarios involving Gaussian noise and modality missing conditions. Moreover, our model exhibits strong generalization capabilities to out-of-distribution data, underscoring its potential as a promising solution for multimodal eye disease screening.

Cross-modal attention network for retinal disease classification based on multi-modal images.

Multi-modal eye disease screening improves diagnostic accuracy by providing lesion information from different sources. However, existing multi-modal automatic diagnosis methods tend to focus on the specificity of modalities and ignore the spatial correlation of images. This paper proposes a novel cross-modal retinal disease diagnosis network (CRD-Net) that digs out the relevant features from modal images aided for multiple retinal disease diagnosis. Specifically, our model introduces a cross-modal attention (CMA) module to query and adaptively pay attention to the relevant features of the lesion in the different modal images. In addition, we also propose multiple loss functions to fuse features with modality correlation and train a multi-modal retinal image classification network to achieve a more accurate diagnosis. Experimental evaluation on three publicly available datasets shows that our CRD-Net outperforms existing single-modal and multi-modal methods, demonstrating its superior performance.

Multimodal (Inter)action Analysis for the study of lectures: active and passive uses of metadiscourse

Abstract This paper offers a Multimodal (Inter)action Analysis study of how metadiscourse is used in university lectures. Metadiscourse is frequently employed in spoken academic discourse to guide the audience through the contents of the speech, thus becoming an essential element to foster comprehension in lectures. Although lectures have been largely researched under a multimodal eye, studies looking at the multimodal nature of metadiscourse are still scarce. In fact, previous multimodal explorations of metadiscourse in lectures point towards discrepancies in the attention given by lecturers to metadiscursive instances. In this study, six face-to-face lectures in fields within Humanities were analyzed to spot all instances of organizational metadiscourse. Next, the fragments containing such metadiscourse were further explored through a Multimodal (Inter)action Analysis to identify the structure of higher-level actions and the ways in which metadiscourse was integrated as part of the modal configurations of the actions. The analysis of higher-level actions using the foreground-background continuum reveals two main roles in the use of metadiscourse: an active one, in which metadiscourse is explicitly used to guide and engage the audience, as expected; and a passive one, in which metadiscourse is rather used as a filler in the background. These results contribute to reflecting on teaching practices and raising awareness on the importance of multimodal literacy for teacher training.

Improving Intention Detection in Single-Trial Classification Through Fusion of EEG and Eye-Tracker Data

Intention decoding is an indispensable procedure in hands-free human–computer interaction (HCI). A conventional eye-tracker system using a single-model fixation duration may issue commands that ignore users' real expectations. Here, an eye-brain hybrid brain–computer interface (BCI) interaction system was introduced for intention detection through the fusion of multimodal eye-tracker and event-related potential (ERP) [a measurement derived from electroencephalography (EEG)] features. Eye-tracking and EEG data were recorded from 64 healthy participants as they performed a 40-min customized free search task of a fixed target icon among 25 icons. The corresponding fixation duration of eye tracking and ERP were extracted. Five previously-validated linear discriminant analysis (LDA)-based classifiers [including regularized LDA, stepwise LDA, Bayesian LDA, shrinkage linear discriminant analysis (SKLDA), and spatial-temporal discriminant analysis] and the widely-used convolutional neural network (CNN) method were adopted to verify the efficacy of feature fusion from both offline and pseudo-online analysis, and the optimal approach was evaluated by modulating the training set and system response duration. Our study demonstrated that the input of multimodal eye tracking and ERP features achieved a superior performance of intention detection in the single-trial classification of active search tasks. Compared with the single-model ERP feature, this new strategy also induced congruent accuracy across classifiers. Moreover, in comparison with other classification methods, we found that SKLDA exhibited a superior performance when fusing features in offline tests (ACC = 0.8783, AUC = 0.9004) and online simulations with various sample amounts and duration lengths. In summary, this study revealed a novel and effective approach for intention classification using an eye-brain hybrid BCI and further supported the real-life application of hands-free HCI in a more precise and stable manner.

Research on athlete's wrong movement prediction method based on multimodal eye movement recognition

Research on athlete's wrong movement prediction method based on multimodal eye movement recognition

Combining Iris, Sclera and Pupil Features for Biometric Authentication System on Smartphone Devices

Currently, biometric authentication systems are commonly used based on physical and behavioural biometric modalities like iris, face, fingerprints, ear, sclera, DNA, voice, signature, etc. Rather than relying on the standalone or unimodal biometric system, multimodal biometric systems are secure and provide more accurate results for person identification and verification. This paper introduces the multimodal eye biometric authentication system where iris, pupil and sclera features are extracted using CNN based on entropy values to perform the accurate automatic segmentation for smartphone devices. The eye images used in the proposed approach for training and testing are completely captured by smartphones. The fusion method used to fuse the colour and texture characteristics of iris and pupil with Y-shaped sclera characteristics from eye image based on support value is Feature Level Fusion. As the images are captured in normal environment settings, it is an unconstrained colour eye image database. MATLAB is used for the experimentation and testing of the model. The proposed eye biometric system outperforms in the case of segmentation and recognition accuracy. Recognition accuracy is –% for unconstrained eye images achieved for the eye image database captured by smartphones.

Multimodal Human Eye Blink Recognition Using Z-score Based Thresholding and Weighted Features

Multimodal Human Eye Blink Recognition Using Z-score Based Thresholding and Weighted Features

Research on athlete's wrong movement prediction method based on multimodal eye movement recognition

Research on athlete's wrong movement prediction method based on multimodal eye movement recognition

Investigating predictive models for earlier diagnosis of cognitive impairment using multimodal eye biomarkers

AbstractBackgroundThe purpose of this study is to investigate predictive models for earlier diagnosis of cognitive impairment (CI) using the eye.MethodProspective age‐matched subjects (n = 69, 55+ years) w/o CI and the presence of any ophthalmic history were recruited. The Montreal Cognitive Assessment scores, retinal images (EasyScan, iOptics) and full‐field electroretinogram (RETevalTM, LKC Technologies, Inc.) were obtained. The multifractal behavior in the skeletonized optic‐disc region was analyzed using the generalized dimensions (D0, D1 & D2) and singularity spectrum f(α) vs. α, both calculated with the ImageJ program. The lacunarity (Λ) was also calculated by measuring the gap dispersion inside each retinal image. Logistic regression was used to construct predictive models to discriminate between phenotypes obtained from individuals w/o CI. Independent variables were divided into sets (see Table 1). Then, five hierarchical set models were fitted with all independent variables forced in. For each model, predicted probabilities were used to construct Receiver Operating Characteristic (ROC) curves. In a separate analysis, all independent variables were allowed inclusion in a parsimonious forward stepwise fashion, and a ROC curve was also constructed with its model‐predicted probabilities. Efficacy of discrimination was summarized with the area under the ROC curve (AUROC) and Youden’s index.ResultOf the 69 participants, 32 had CI (46%). Figures 1 and Table 1 shows that the overall predictive accuracy of the model 5 in discriminating patients with CI from cognitive healthy subjects may be better (AUROC∼0.95) than that of the other combined measurements AUROC range∼[0.73 ‐ 0.88]. In the separated analysis with all independent variables, the singularity exponent a2 was the most significant predictor of CI. Once this was accounted for, none of the other parameters was statistically significant except Flicker IT. Therefore, a2 and Flicker IT were included in a single model to obtain a powerful predictive index: Xlinear = 18.387 + (0.736) × (Flicker IT)‐(26.887) x (a2) being the Predictive probability of CI = expXlinear/(1+expXlinear). The AUROC for this predictive model was 0.897 (SE = 0.050) and was highly significant (p < 0.001).ConclusionOur results showed that the predictive model using multimodal eye biomarkers has potential to target cognitive screening toward individuals at increased risk of CI.

Multimodal human eye blink recognition method using feature level fusion for exigency detection

In this paper, a precise multimodal eye blink recognition method using feature level fusion (MmERMFLF) is proposed. A new feature: eye-eyebrow facet ratio (EEBFR) (formed by fusing eye facet ratio (EFR: ratio of diagonal length and width of eye) and eyebrow to nose facet ratio (EBNFR: distance between eyebrow landmarks and nose landmark)) for approximating the eye state is computed. Initially, an improved intellectual framework (Sagacious Information Recuperation Technique) that senses the emergency state using information retrieved from eye blinks, pulse rate as well as behavioral patterns(emotions) exhibited by an individual is presented. Further a novel multimodal method (MmERMFLF) for detection and counting of eye blinks is implemented. For training, one state-of-the-art database—ZJU is used. To additionally improve the performance, feature-level fusion schemes [simple concatenate and fusion codes (gaborization)] are enforced and equated. Receiver operating characteristics, error rate, sensitivity, specificity, and precision are used to demonstrate the performance of the proposed method qualitatively and quantitatively. Accuracy with proposed MmERMFLF is increased to 99.02% (using EEBGFR method with bagged ensemble classifier) in comparison to unimodal eye blink recognition system (97.60%). 99.80% genuine blinks are classified by MmERMFLF (when gaborization fusion is used) using simple tree classifier.

A Multimodal Eye Assessment in Psoriatic Arthritis Patients sine-Psoriasis: Evidence for a Potential Association with Systemic Inflammation.

Background: Ocular involvement in Psoriatic Arthritis (PsA) patients is mainly associated with uveitis but there remains a paucity of data on dry eye and retinal abnormalities. We aimed to analyze dry eye and subclinical retinal abnormalities in a cohort of PsA patients sine-psoriasis (PsO). Methods: PsA patients sine-PsO were enrolled. Best-corrected-visual-acuity, ocular-surface-disease-index (OSDI), Schirmer test, tear film breakup-time, standard-automated-perimetry (SAP, mean deviation—MD, pattern standard deviation—PSD), fundus-perimetry (FP), and spectral-domain-optical-coherence-tomography (SD-OCT) were performed. Results: A total of 80 eyes from 40 PsA patients with moderate-severe disease activity, and 70 eyes from 35 healthy control (HC) were evaluated. Higher dry eye prevalence occurred in PsA than HC (p < 0.0001). ESR was positively related with OSDI (p < 0.001) and negatively related with Schirmer (p = 0.007). In PsA, SAP registered higher MD (p < 0.0001) and higher PSD (p = 0.005) in comparison with HC. PSD resulted positively correlated with ESR (p = 0.04) and CRP (p = 0.01), while MD showed a negative correlation with CRP (p = 0.01). Both FP mean differential sensitivity and mean defect were lower in PsA then HC (p < 0.0001). In PsA, FP differential sensitivity was directly related with cumulative steroids (p = 0.02). Conclusions: In PsA patients sine-PsO, dry eye and subclinical abnormalities in visual functions occurred being potentially related to systemic inflammation.

INVESTIGATING DIAGNOSTIC BIOMARKERS OF COGNITIVE IMPAIRMENT BY UTILIZING VASCULAR AND NEUROGENIC CHANGES IN THE RETINA

Previous studies have demonstrated that Alzheimer's disease (AD) it is not limited to the brain but also affects the retina. We investigated the correlation between the retinal vascular complexity and neurodegenerative changes in patients with mild cognitive impairment (MCI). Quantification of the retinal structure and function using advanced retinal imaging and full-field electroretinogram (ERG) was performed. Also, visual performance exams were conducted using the Rabin cone contrast test (CCT). Retinal images and ERGs from each subject were obtained. Retinal vascular parameters were calculated using the Singapore Institute Vessel Assessment (SIVA) software which was also used to calculate the fractal dimension (FD) of the retinal vasculature network and retinal vascular caliber, summarized as central retinal artery equivalent (CRAE) and central retinal vein equivalent (CRVE). The Montreal Cognitive Assessment (MoCA) was used to assess cognitive dysfunction. Spearman's linear correlation was performed between variables, the level of significance was set at 5%. For the analysis, Statistics 7.0 software was used (Statsoft Inc., Tulsa, OK). Of the 55 participants, 21 had cognitive impairment (38%), 34 served as healthy controls. We found that the complexity of the branching pattern and caliber of retinal vessels were lower in patients with MCI (FD (mean±SE): 1.310±0.020; CRVE (92.65±1.60; age range [76-86]) in comparison to healthy controls (FD: 1.382±0.002; CRVE: 202.15±0.76; age range [76-86]). Also, drusen and retinal pigment mottling were noted in some elderly MCI subjects. For all full-field ERG measurements, smaller amplitudes and larger peak times were observed in the MCI subjects. The visual performance test with the computerized Rabin CCT revealed functional loss in color vision. Interestingly, FD significantly correlated with CRVE (p=0.0212, r=-0.4628) and with the ERG's implicit time (p=0.0240, r=0.4910) in MCI individuals. To our knowledge, this is the first study to examine the association between FD and full field ERG implicit time in patients with MCI. Our preliminary findings also show that our approach to evaluating visual capacities in elderly individuals may add predictive value of early visual pathway injury associated with MCI and facilitate the introduction of a novel multimodal eye biomarker for the early detection of AD.

A comprehensive multimodal eye recognition

Iris recognition is tested to be one of the most reliable approaches for automatic personal recognition under the infrared light. However, it is challenging and difficult to acquire good quality iris patterns of dark color eyes in the visible wavelengths. Sclera recognition can achieve good recognition accuracy in the visible spectrum, but the performance will drop when the sclera pattern is saturated. In this paper, we proposed a comprehensive multimodal eye recognition system that uses both iris and sclera patterns for recognition from the same eye image. The experimental results show that the proposed multimodal frontal eye recognition method can achieve better recognition accuracy than unimodal iris or sclera recognition. In addition, we propose a multimodal multi-angle eye recognition system that can further improve the recognition accuracy.

Automated Assessment of Diabetic Retinopathy Severity Using Content-Based Image Retrieval in Multimodal Fundus Photographs

Recent studies on diabetic retinopathy (DR) screening in fundus photographs suggest that disagreements between algorithms and clinicians are now comparable to disagreements among clinicians. The purpose of this study is to (1) determine whether this observation also holds for automated DR severity assessment algorithms, and (2) show the interest of such algorithms in clinical practice. A dataset of 85 consecutive DR examinations (168 eyes, 1176 multimodal eye fundus photographs) was collected at Brest University Hospital (Brest, France). Two clinicians with different experience levels determined DR severity in each eye, according to the International Clinical Diabetic Retinopathy Disease Severity (ICDRS) scale. Based on Cohen's kappa (κ) measurements, the performance of clinicians at assessing DR severity was compared to the performance of state-of-the-art content-based image retrieval (CBIR) algorithms from our group. At assessing DR severity in each patient, intraobserver agreement was κ = 0.769 for the most experienced clinician. Interobserver agreement between clinicians was κ = 0.526. Interobserver agreement between the most experienced clinicians and the most advanced algorithm was κ = 0.592. Besides, the most advanced algorithm was often able to predict agreements and disagreements between clinicians. Automated DR severity assessment algorithms, trained to imitate experienced clinicians, can be used to predict when young clinicians would agree or disagree with their more experienced fellow members. Such algorithms may thus be used in clinical practice to help validate or invalidate their diagnoses. CBIR algorithms, in particular, may also be used for pooling diagnostic knowledge among peers, with applications in training and coordination of clinicians' prescriptions.