Visual Domain Research Articles

Event-related potentials of stimuli inhibition and access in cross-modal distractor-induced blindness.

Distractor-induced blindness (DIB) describes a reduced access to a cued visual target-if multiple target-like distractors have been presented beforehand. Previous ERP data suggest a cumulative frontal inhibition triggered by distractors, which affects the updating process of the upcoming target. In the present study, we examine whether the modality of the cue-formerly defined in the visual domain-affects the expression of these neural signatures. 27 subjects were tested in a cross-modal DIB task: Distractors and targets were defined by a transient change of stimuli shape in a random-dot kinematogram. The onset of the target was announced by a rise in amplitude of a sinusoidal tone. Behavioral results confirmed that detection of the target relies on the number of preceding distractor episodes. Replicating previous unimodal results, ERP responses to distractors were characterized by a frontal negativity starting at 100 ms, which increases with an increasing number of distractor episodes. However, the processing-and detection-of the target was not characterized by a more-expressed P3 response, but by an occipital negativity. The current data confirm that the neural signatures of target awareness depend on the experimental setup used: In case of the DIB, the cross-modal setting might lead to a reduction of attentional resources in the visual domain.

The Neural Specificity of Interference Resolution in Phonological, Semantic, and Visual Domains at Different Ages.

The question of whether cognitive control is specific to certain domains or domain-general remains an extensively debated question at both cognitive and neural levels. This study examined the neural substrates associated with resistance to interference (RI) in phonological, semantic, and visual domains by using strictly matched tasks and determining the domain-general or domain-specific manner in which aging affects the neural substrates associated with RI. In an fMRI experiment, young and older participants performed a similarity judgment task with phonological, semantic, or visual interference buildup. For both age groups, domain-specific RI effects were observed at the univariate level, with increased involvement in the phonological domain of the right angular gyrus and the right lingual gyrus, in the semantic domain of the bilateral inferior frontal gyrus, the bilateral superior parietal and angular gyri and the left middle temporal gyrus, and in the visual domain of the middle/superior frontal gyri and occipital gyri. At the multivariate level, although RI effects could be decoded from neural patterns in the bilateral inferior frontal gyrus for all domains and age groups, between-domain prediction of RI conditions was associated with Bayesian evidence for the null hypothesis. This study supports the domain specificity of neural substrates associated with RI while stressing its age independency.

Orthrus: Towards Evolutionary and Functional RNA Foundation Models.

In the face of rapidly accumulating genomic data, our understanding of the RNA regulatory code remains incomplete. Pre-trained genomic foundation models offer an avenue to adapt learned RNA representations to biological prediction tasks. However, existing genomic foundation models are trained using strategies borrowed from textual or visual domains, such as masked language modelling or next token prediction, that do not leverage biological domain knowledge. Here, we introduce Orthrus, a Mamba-based RNA foundation model pre-trained using a novel self-supervised contrastive learning objective with biological augmentations. Orthrus is trained by maximizing embedding similarity between curated pairs of RNA transcripts, where pairs are formed from splice isoforms of 10 model organisms and transcripts from orthologous genes in 400+ mammalian species from the Zoonomia Project. This training objective results in a latent representation that clusters RNA sequences with functional and evolutionary similarities. We find that the generalized mature RNA isoform representations learned by Orthrus significantly outperform existing genomic foundation models on five mRNA property prediction tasks, and requires only a fraction of fine-tuning data to do so.

LLM-enhanced Composed Image Retrieval: An Intent Uncertainty-aware Linguistic-Visual Dual Channel Matching Model

Composed image retrieval (CoIR) involves a multi-modal query of the reference image and modification text describing the desired changes, allowing users to express image retrieval intents flexibly and effectively. The key of CoIR lies in how to properly reason the search intent from the multi-modal query. Existing work either aligns the composite embedding of the multi-modal query and the target image embedding in the visual domain through late-fusion or converts all images into text descriptions and leverage large language models (LLM) for text semantic reasoning. However, this single-modality reasoning approach fails to comprehensively and interpretably capture the users’ ambiguous and uncertain intents in the multi-modal queries, incurring the inconsistency between retrieved results and ground truth. Besides, the expensive manually-annotated datasets limit the further performance improvement of CoIR. To this end, this paper proposes an LLM-enhanced Intent Uncertainty-aware Linguistic-Visual Dual Channel Matching Model (IUDC), which combines the strengths of multi-modal late-fusion and LLMs for composed image retrieval. We first construct an LLM-based triplet augmentation strategy to generate more synthetic training triplets. Based on this, the core of IUDC consists of two matching channels: the semantic matching channel is responsible for intent reasoning on the aspect-level attributes extracted by an LLM, and the visual matching channel accounts for the fine-grained visual matching between multi-modal fusion embedding and target images. Considering the intent uncertainty presented in the multi-modal queries, we introduce Probability Distribution Encoder (PDE) to project the intents as probabilistic distributions in the two matching channels. Consequently, a mutually enhanced module is designed to share knowledge between the visual and semantic representations for better representation learning. Finally, the matching scores of two channels are added to retrieve the target image. Extensive experiments conducted on two real datasets demonstrate the effectiveness and superiority of our model. Notably, with the help of the proposed LLM-based triplet augmentation strategy, our model achieves a new record of state-of-the-art performance among all datasets.

Visual placebo and nocebo effects.

Placebo and nocebo effects modulate symptom perception through expectations and learning processes in various domains. Predominantly, their impact has been investigated on pain and physical performance. However, the influence of placebos and nocebos on visual system functionality has yet to be explored. The present study aimed to test whether placebo and nocebo effects can intervene in altering participants' performance outcomes during a novel visual accuracy task and to examine the underlying neural mechanisms through EEG. After performing a baseline session, visual accuracy was said to be enhanced or disrupted by a sham transcranial electrical stimulation over the occipital lobe. Behavioural results showed a significant increase in visual accuracy for the placebo group, from the baseline session to the test session, whereas the nocebo group showed a decrease in visual accuracy. EEG analyses on the event-related potential P300 component, conducted on both a centro-parietal electrode patch and a parieto-occipital, one displayed an increase in the amplitude of P300 for the placebo group, and a decrease in the nocebo group. These findings suggest for the first time that placebo and nocebo effects can influence visual perception and attentional processes linked to it. Overall, the present study contributes to understanding how expectations affect sensory perception beyond pain and the motor system, paving the way for investigating these phenomena in other sensory modalities such as auditory or olfactory perception. KEY POINTS: Placebo and nocebo effects have been studied predominantly in pain and motor performance fields. In a novel visual task, the impact of placebo and nocebo effects on the visual system has been evaluated, in both early components (stimuli-related) and late components (attention-related). The placebo group showed an increase in visual accuracy and EEG-evoked potential amplitudes, whereas the nocebo group showed a decrease in both. This study shows how expectations and the related placebo and nocebo effects can shape basic stimuli sensory perception in the visual domain.

How does visual context of urban open spaces drive soundscape pleasantness and eventfulness: a laboratory experiment using a higher order Ambisonics speaker system and immersive virtual reality

Perception of sound environments is influenced by the context we experience them in. A big portion of the contextual information comes from the visual domain, including our understanding of what a place is but also from its visual features. A laboratory study combining questionnaire responses and eye-tracking tools was designed to investigate if the soundscape outcomes and participants' behaviour inside the simulation can be explained by the perceptual outcomes defined by visual information. 360 degree videos and First Order Ambisonics audio recordings of 27 different urban open spaces taken from the International Soundscape Database were used as the stimuli delivered via a Virtual Reality Head-Mounted Display and a Higher Order Ambisonics speaker array, while a neutral grey environment without sounds being reproduced represented the baseline scenario. A questionnaire tool was deployed within the IVR simulation to collect participants' responses describing their perception of the reproduced environments corresponding to the circumplex model featured in the Method A of the ISO/TS 12913-2. The results revealed a good coverage of the two-dimensional perceptual circumplex space and significant differences between perceptual outcomes driven by sound and those driven by visual stimuli.

Monitoring of indoor environmental quality and acoustic comfort in university classrooms

Indoor Environmental Quality (IEQ) in university classrooms affects students' well-being, comfort and learning performance, thus its evaluation is strongly needed. This work presents an in-field study on the acoustic monitoring, along with thermal, visual and air quality domains, in four recently built university classrooms of Politecnico di Torino. The monitoring was performed in the spring/summer and winter periods, through wall mounted multi-sensor devices. While the monitoring was ongoing, students attending lessons were asked to answer an online anonymous questionnaire on comfort perception, which included questions regarding the four domains and additional personal information. Results show that students are more satisfied with the overall conditions in classroom 3P and more dissatisfied with the overall conditions in classroom 4P. The less satisfying domains are the thermal and air quality ones, whereas the more satisfying are the acoustic and visual ones. Furthermore, at the increasing of TVOC and CO2 concentrations the overall satisfaction decreases. The final aim of this project, based on the monitoring results, is the optimization of the building systems management to reduce energy consumptions and ensure occupants' well-being and comfort.

SecureVision: Advanced Cybersecurity Deepfake Detection with Big Data Analytics

SecureVision is an advanced and trustworthy deepfake detection system created to tackle the growing threat of ‘deepfake’ movies that tamper with media, undermine public trust, and jeopardize cybersecurity. We present a novel approach that combines big data analytics with state-of-the-art deep learning algorithms to detect altered information in both audio and visual domains. One of SecureVision’s primary innovations is the use of multi-modal analysis, which improves detection capabilities by concurrently analyzing many media forms and strengthening resistance against advanced deepfake techniques. The system’s efficacy is further enhanced by its capacity to manage large datasets and integrate self-supervised learning, which guarantees its flexibility in the ever-changing field of digital deception. In the end, this study helps to protect digital integrity by providing a proactive, scalable, and efficient defense against the ubiquitous threat of deepfakes, thereby establishing a new benchmark for privacy and security measures in the digital era.

Artistic Expertise and Free Viewing of Modern Art

Abstract Visual art and vision science had long been intertwined, both revealing and sharing countless insights on human vision over their respective histories. Deep artistic experiences and extensive artistic knowledge are often associated with “artistic expertise,” which in vision science traditionally echoes visual expertise from other professional domains (e.g., medicine, sports, driving). However, unlike other visual domains, artistic experiences involve more spontaneous and imaginative aspects, and no explicit success parameters. We hypothesized that viewing of artworks would lead to nuanced effects of expertise that may differ from other types of visual expertise. In our study, art experts and non-experts observed modern paintings and were free to view them within a relatively long time window. Higher expertise in art was associated with viewing modern paintings for significantly longer durations, and liking the paintings significantly more. However, contrary to other visual expertise studies, eye movement patterns exhibited little to no effect of artistic expertise. Based on these findings, we suggest a recontextualization of visual artistic expertise and discuss potential research directions, alongside methodological challenges in the experimental design.

Zero-shot visual grounding via coarse-to-fine representation learning

Visual grounding (VG) locates target objects in visual scenes by understanding given natural language queries. Current methods for VG mainly focus on grounding referring expressions or noun phrases covered in the labeled training samples. Despite their grounding prowess, these approaches struggle in grounding novel query–image pairs excluded from the training data. This shortage is usually caused by the deficiency of discriminative representation learning both in images and queries. To address these issues, we propose a one-stage coarse-to-fine framework for zero-shot VG to ground novel query-image samples. Specifically, in the coarse stage, we mine the global context information in the visual features and query embeddings by employing a multi-head self-attention block, strengthening the intra-modality relations in the visual and textual features. In the fine stage, we first learn the query-aware visual representations based on the acquired global context information via a multi-modal relation-enhanced transformer block, which explores the informative information by modeling the cross-modal interaction among visual and textual domains. We further excavate target-oriented discriminative representations from the acquired query-aware visual representations by a noun phrase-guided multi-modal interaction network, which augments the interaction between target-related phrases and the obtained query-aware visual representations to enhance the distinction of target regions, enhancing the subsequent referred target grounding and generalizing. In order to validate the proposed approach, we implement extensive experiments and ablation studies on public benchmark datasets, including RefCOCO, RefCOCO+, RefCOCOg, Flickr30K Entity, Flickr-Split-0 and Flickr-Split-1. Experimental results demonstrate that our approach substantially improves the grounding accuracy and achieves new state-of-the-art performance under single-stage training and testing.

The Authenticity of Diriyah: Between the Philosophy of Place and Time as a Foundation for Designing Contemporary Artwork

This study delves into the integration of the philosophical concepts of space and time within the context of Diriyah's rich cultural and historical backdrop, aiming to leverage these concepts as foundational pillars for innovative artistic design within the visual arts and design domain. It adopts a hybrid methodology that combines descriptive and analytical techniques for its theoretical framework with experimental approaches for practical application. The outcomes underscore the feasibility and potential of melding space and time philosophies to foster unique design ideas for installation art, thereby enriching the visual arts and design landscape. Some recommendations were presented including engaging deeply with philosophical themes such as the essence of space and time. This research not only contributes to the artistic valorization of Diriyah's heritage within the framework of Saudi Arabia's Vision 2030 but also propels forward the discourse on integrating architectural heritage with contemporary art practices.

B - 31 Factor Analysis of Neuropsychological Attention Measures with Mild Traumatic Brain Injury Patients

Abstract Objective Mild traumatic brain injury (mTBI) often leads to attentional deficits, yet the neuropsychological tests traditionally used may not adequately reflect the cognitive impairments these patients experience. This study aimed to critically evaluate the construct validity and factor structure of attention-related neuropsychological tests in mTBI patients to better understand and diagnose attentional deficits. Method The research utilized a battery of neuropsychological tests used to assess attention among a sample of 269 adults with mTBI including the NAB Numbers & Letters Test, PASAT, Trail Making Test, and WAIS Digit Span, Coding, and Arithmetic subtests. Exploratory factor analysis was employed to investigate the factor structure of the test battery and to test the hypothesis concerning the dimensionality of attention constructs. Results The analysis revealed that attentional measures did not converge on a single latent factor but instead reflected four distinct factors. These factors were identified as 1) Visual Attention/Psychomotor Speed, 2) Auditory Working Memory, 3) Visuospatial Scanning/Search, and 4) Disruption. Conclusions This exploratory factor analysis unveiled a distinction between auditory and visual attention mechanisms within this population. This separation suggests that attentional processes are not monolithic but rather encompass distinct auditory and visual domains and corroborates the existing literature, including Mirsky’s (1991) model of attention, which posits multiple components of attention, albeit with a focus on differentiating processes rather than sensory modalities. The study highlights the complexity of attentional impairments in mTBI and underscores the need for a nuanced approach to the neuropsychological assessment of attention among mTBI patients.

Commonalities and variations in emotion representation across modalities and brain regions

Humans express emotions through various modalities such as facial expressions and natural language. However, the relationships between emotions expressed through different modalities and their correlations with neural activities remain uncertain. Here, we aimed to unveil some of these uncertainties by investigating the similarity of emotion representations across modalities and brain regions. First, we represented various emotion categories as multi-dimensional vectors derived from visual (face), linguistic, and visio-linguistic data, and used representational similarity analysis to compare these modalities. Second, we examined the linear transferability of emotion representation from other modalities to the visual modality. Third, we compared the representational structure derived in the first step with those from brain activities across 360 regions. Our findings revealed that emotion representations share commonalities across modalities with modality-type dependent variations, and they can be linearly mapped from other modalities to the visual modality. Additionally, emotion representations in uni-modalities showed relatively higher similarity with specific brain regions, while multi-modal emotion representation was most similar to representations across the entire brain region. These findings suggest that emotional experiences are represented differently across various brain regions with varying degrees of similarity to different modality types, and that they may be multi-modally conveyable in visual and linguistic domains.

Introducing ART: A new method for testing auditory memory with circular reproduction tasks.

Theories of visual working memory have seen significant progress through the use of continuous reproduction tasks. However, these tasks have mainly focused on studying visual features, with limited examples existing in the auditory domain. Therefore, it is unknown to what extent newly developed memory models reflect domain-general limitations or are specific to the visual domain. To address this gap, we developed a novel methodology: the Auditory Reproduction Task (ART). This task utilizes Shepard tones, which create an infinite rising or falling tone illusion by dissecting pitch chroma and height, to create a 1-360° auditory circular space. In Experiment 1, we validated the perceptual circularity and uniformity of this auditory stimulus space. In Experiment 2, we demonstrated that auditory working memory shows similar set size effects to visual working memory-report error increased at a set size of 2 relative to 1, caused by swap errors. In Experiment 3, we tested the validity of ART by correlating reproduction errors with commonly used auditory and visual working memory tasks. Analyses revealed that ART errors were significantly correlated with performance in both auditory and visual working memory tasks, albeit with a stronger correlation observed with auditory working memory. While these experiments have only scratched the surface of the theoretical and computational constraints on auditory working memory, they provide a valuable proof of concept for ART. Further research with ART has the potential to deepen our understanding of auditory working memory, as well as to explore the extent to which existing models are tapping into domain-general constraints.

Anomaly Detection in Sound Activity with Generative Adversarial Network Models

In state-of-art anomaly detection research, prevailing methodologies predominantly employ Generative Adversarial Networks and Autoencoders for image-based applications. Despite the efficacy demonstrated in the visual domain, there remains a notable dearth of studies showcasing the application of these architectures in anomaly detection within the sound domain. This paper introduces tailored adaptations of cutting-edge architectures for anomaly detection in audio and conducts a comprehensive comparative analysis to substantiate the viability of this novel approach. The evaluation is performed on the DCASE 2020 dataset, encompassing over 180 hours of industrial machinery sound recordings. Our results indicate superior anomaly classification, with an average Area Under the Curve (AUC) of 88.16% and partial AUC of 78.05%, surpassing the performance of established baselines. This study not only extends the applicability of advanced architectures to the audio domain but also establishes their effectiveness in the challenging context of industrial sound anomaly detection.

Deep Learning for Visual Speech Analysis: A Survey.

Visual speech, referring to the visual domain of speech, has attracted increasing attention due to its wide applications, such as public security, medical treatment, military defense, and film entertainment. As a powerful AI strategy, deep learning techniques have extensively promoted the development of visual speech learning. Over the past five years, numerous deep learning based methods have been proposed to address various problems in this area, especially automatic visual speech recognition and generation. To push forward future research on visual speech, this paper will present a comprehensive review of recent progress in deep learning methods on visual speech analysis. We cover different aspects of visual speech, including fundamental problems, challenges, benchmark datasets, a taxonomy of existing methods, and state-of-the-art performance. Besides, we also identify gaps in current research and discuss inspiring future research directions.

Examining the Effectiveness of Speech and Earcon Alerts for Aiding Phishing Email Detection

As technology advances, phishing attacks grow more vicious. Anti-phishing research has primarily focused on visual cues like domain names and contents, leaving visually impaired individuals particularly vulnerable. Recent studies suggest incorporating auditory alerts to enhance the detection of phishing threats. As a first step toward designing more inclusive phishing alerts, the current study examines the effectiveness of two types of audio alerts on individuals with normal or corrected-to-normal vision. Participants were randomly assigned to three alert conditions: control, earcon, and speech alert. Each participant went through 20 emails to indicate whether each email was legitimate. The results showed that participants in the two alert groups spent less time on each email than the control. Additionally, participants in the earcon condition showed a greater tendency to classify emails as phishing than those in the control condition. These results can inform the future development of auditory alerts for phishing.

Implicit audiomotor adaptation

Sensorimotor adaptation alters mappings between motor commands and their predicted outcomes. Such remapping has been extensively studied in the visual domain, but the degree to which it occurs in modalities other than vision remains less well understood. Here, we manipulated the modality of reach target presentation to compare sensorimotor adaptation in response to perturbations of visual and auditory feedback location. We compared the extent of adaptation to perturbed sensory feedback for visual and auditory sensory modalities, and the magnitude of reach-direction aftereffects when the perturbation was removed. To isolate the contribution of implicit sensorimotor recalibration to adaptation in reach direction, we held sensory prediction errors and task-performance errors constant via a task-irrelevant clamp of sensory feedback. Seventy-two participants performed one of three experiments in which target location information and endpoint reach direction feedback were presented by loudspeakers (n = 24), headphones (n = 24), or a visual display (n = 24). Presentation durations for target stimuli (500 ms) and (non-veridical) endpoint feedback of reach direction (100 ms) were matched for visual and auditory modalities. For all three groups, when endpoint feedback was perturbed, adaptation was evident: reach-directions increased significantly in the direction opposite the clamped feedback, and a significant aftereffect persisted after participants were instructed that the perturbation had been removed. This study provides new evidence that implicit sensorimotor adaptation occurs in response to perturbed auditory feedback of reach direction, suggesting that an implicit neural process to recalibrate sensory to motor maps in response to sensory prediction errors may be ubiquitous across sensory modalities.

Clustering of cognitive domains among older adults with HIV

Age and HIV are synergistic risk factors for conditions such as HIV-associated neurocognitive disorders (HAND). Yet, it is unclear whether OAWH display different cognitive profiles for HAND. To describe the cognitive patterns of OAWH treated with combined antiretroviral therapy (cART). Cross-sectional study that included 330 participants with HIV, aged 50 years or older, cared for at a tertiary care hospital in Mexico City. A short neuropsychological test battery was used to assess a wide spectrum of cognitive functions. The optimal number of cognitive clusters was determined by the silhouette method and a minimization of the Bayesian information criterion. Participants' mean age was 58.8 years (SD = 6.6), and 12.1% were women. A 3-cluster solution yielded stable Jaccard coefficients (p > 0.70). Cluster 1 showed more significant impairment in visual and verbal memory domains, whereas participants in cluster 3 showed significant impairment in language, and abstraction. Cluster 2 showed no predominance of any domain for alterations. There are different cognitive profiles among OAWH with HAND. These differences may be due to individual patterns of HIV-related and non-HIV-related factors.

Cross-modal domain generalization semantic segmentation based on fusion features

The primary techniques for domain generalization in semantic segmentation revolve around domain randomization and feature whitening. Although less commonly employed, methods based on cross-modality have demonstrated effective outcomes. This paper introduces enhancements to cross-modal feature alignment by redesigning the feature alignment module. This redesign facilitates alignment across different modalities by leveraging fusion features derived from both visual and textual inputs. These fusion features provide a more effective anchor point for alignment, enhancing the transfer of semantic information from textual to visual domains. Furthermore, the decoder plays a crucial role in the model as its ability to categorize features directly impacts the segmentation performance of the entire model. To enhance the decoder’s capability, this study employs the fusion features as the input for the decoder, with image labels providing the supervision. Experimental results indicate that our approach significantly enhances the model’s generalization capabilities.