Modulation Of Visual Attention Research Articles

Auditory change detection and visual selective attention: association between MMN and N2pc.

While the auditory and visual systems each provide distinct information to our brain, they also work together to process and prioritize input to address ever-changing conditions. Previous studies highlighted the trade-off between auditory change detection and visual selective attention; however, the relationship between them is still unclear. Here, we recorded electroencephalography signals from 106 healthy adults in three experiments. Our findings revealed a positive correlation at the population level between the amplitudes of event-related potential indices associated with auditory change detection (mismatch negativity) and visual selective attention (posterior contralateral N2) when elicited in separate tasks. This correlation persisted even when participants performed a visual task while disregarding simultaneous auditory stimuli. Interestingly, as visual attention demand increased, participants whose posterior contralateral N2 amplitude increased the most exhibited the largest reduction in mismatch negativity, suggesting a within-subject trade-off between the two processes. Taken together, our results suggest an intimate relationship and potential shared mechanism between auditory change detection and visual selective attention. We liken this to a total capacity limit that varies between individuals, which could drive correlated individual differences in auditory change detection and visual selective attention, and also within-subject competition between the two, with task-based modulation of visual attention causing within-participant decrease in auditory change detection sensitivity.

The Interaction of Linguistic and Visual Cues for the Processing of Case in Russian by Russian-German Bilinguals: An Eye Tracking Study.

Modulation of visual attention in the Visual World Paradigm relies on parallel processing of linguistic and visual information. Previous studies have argued that the human linguistic capacity includes an aspect of anticipation of upcoming material. Such anticipation can be triggered by both lexical and grammatical/morphosyntactic cues. In this study, we investigated the relationship between comprehension and prediction by testing how subtle changes in visual representations can affect the processing of grammatical case cues in Russian by Russian-German bilingual children (n = 49, age 8-13). The linguistic manipulation followed previous designs, contrasting SVO and OVS sentences, where the first NP (NP1) was marked with nominative or accusative case, respectively. Three types of visual displays were compared: (i) individual referents (potential agent/theme); (ii) pairs of referents (NP1 + potential agent/theme); and (iii) events (representing interactions between the referents). Participants were significantly more sensitive to the case manipulation when presented with events compared to the other two types of visual display. This suggests that they were able to quickly integrate the thematic role information signaled by grammatical case in the event representations. However, they were less likely to use the case information to anticipate upcoming arguments when the target pictures represented individual referents or pairs of noninteracting referents. We hypothesize that the process of argument anticipation is mediated by the activation of syntactic templates (SVO or OSV, depending on the case marking on NP1). The relatively weak anticipation effect observed may be attributed to the absence, or weak representation, of the noncanonical OVS template in the bilingual children's long-term memory.

Multimodal hate speech detection via multi-scale visual kernels and knowledge distillation architecture

People increasingly use social media platforms to express themselves by posting visuals and texts. As a result, hate content is on the rise, necessitating practical visual caption analysis. Thus, the relationship between image and caption modalities is crucial in visual caption analysis. Contrarily, most methods combine features from the image and caption modalities using deep learning architectures with millions of parameters already trained without integrating a specialized attention module, resulting in less desirable outcomes. This paper suggests a novel multi-modal architecture for identifying hateful memetic information in response to the above observation. The proposed architecture contains a novel "multi-scale kernel attentive visual" (MSKAV) module that uses an efficient multi-branch structure to extract discriminative visual features. Additionally, MSKAV utilizes an adaptive receptive field using multi-scale kernels. MSKAV also incorporates a multi-directional visual attention module to highlight spatial regions of importance. The proposed model also contains a novel "knowledge distillation-based attentional caption" (KDAC) module. It uses a transformer-based self-attentive block to extract discriminative features from meme captions. Thorough experimentation on multi-modal hate speech benchmarks MultiOff, Hateful Memes, and MMHS150K datasets achieved accuracy scores of 0.6250, 0.8750, and 0.8078, respectively. It also reaches impressive AUC scores of 0.6557, 0.8363, and 0.7665 on the three datasets, respectively, beating SOTA multi-modal hate speech identification models.

TasselLFANet: a novel lightweight multi-branch feature aggregation neural network for high-throughput image-based maize tassels detection and counting.

Accurately and rapidly counting the number of maize tassels is critical for maize breeding, management, and monitoring the growth stage of maize plants. With the advent of high-throughput phenotyping platforms and the availability of large-scale datasets, there is a pressing need to automate this task for genotype and phenotype analysis. Computer vision technology has been increasingly applied in plant science, offering a promising solution for automated monitoring of a large number of plants. However, the current state-of-the-art image algorithms are hindered by hardware limitations, which compromise the balance between algorithmic capacity, running speed, and overall performance, making it difficult to apply them in real-time sensing field environments. Thus, we propose a novel lightweight neural network, named TasselLFANet, with an efficient and powerful structure for accurately and efficiently detecting and counting maize tassels in high spatiotemporal image sequences. Our proposed approach improves the feature-learning ability of TasselLFANet by adopting a cross-stage fusion strategy that balances the variability of different layers. Additionally, TasselLFANet utilizes multiple receptive fields to capture diverse feature representations, and incorporates an innovative visual channel attention module to detect and capture features more flexibly and precisely. We conducted a series of comparative experiments on a new, highly informative dataset called MrMT, which demonstrate that TasselLFANet outperforms the latest batch of lightweight networks in terms of performance, flexibility, and adaptability, achieving an F1 measure value of 94.4%, a mAP.@5 value of 96.8%, and having only 6.0M parameters. Moreover, compared with the regression-based TasselNetV3-Seg† model, our proposed model achieves superior counting performance, with a mean absolute error (MAE) of 1.80, a root mean square error (RMSE) of 2.68, and a R2 of 0.99. The proposed model meets the accuracy and speed requirements of the vision system in maize tassel detection. Furthermore, our proposed method is reliable and unaffected by geographical changes, providing essential technical support for computerized counting in the field.

No evidence for spatial suppression due to across-trial distractor learning in visual search

Previous studies have shown that during visual search, participants are able to implicitly learn across-trial regularities regarding target locations and use these to improve search performance. The present study asks whether such across-trial visual statistical learning also extends to the location of salient distractors. In Experiments 1 and 2, distractor regularities were paired so that a specific distractor location was 100% predictive of another specific distractor location on the next trial. Unlike previous findings that employed target regularities, the current results show no difference in search times between predictable and unpredictable trials. In Experiments 3–5 the distractor location was presented in a structured order (a sequence) for one group of participants, while it was presented randomly for the other group. Again, there was no learning effect of the across-trial regularities regarding the salient distractor locations. Across five experiments, we demonstrated that participants were unable to exploit across-trial spatial regularities regarding the salient distractors. These findings point to important boundary conditions for the modulation of visual attention by statistical regularities and they highlight the need to differentiate between different types of statistical regularities.

Electromagnetic Imaging Boosted Visual Object Recognition Under Difficult Visual Conditions

Object imaging and recognition under difficult visual conditions is extremely challenging due to the captured low-quality images, and traditional optical-based recognition methods always fail in this task. In this paper, we propose to utilize the visual-microwave image pairs captured by both visual cameras and microwave sensors for imaging and recognition. To address the heavy noises in the low-quality optical images, we retrieve the physically quantitative images from associated scattered field data, and enhance visual features by both optical and retrieval images. We develop a cross-modal Enhanced Attentive Visual-Microwave Fusion (EAVMF) object recognition model to jointly learn the cross-modal generator and multimodal recognizer. In addition, an attention module for the visual subnetwork is utilized to highlight the regions of interest. Two multimodal datasets with synthetic visual-microwave image pairs are built to simulate the difficult visual condition. The numerical results on these datasets demonstrate that: 1) both the multimodal fusion, cross-modal enhancement, and visual attention module can enhance the performance; and 2) compared with existing methods, the proposed EAVMF not only performs better in terms of accuracy but also has good scalability and one-shot learning ability.

An improved anchor-free SAR ship detection algorithm based on brain-inspired attention mechanism.

As a computing platform that can deal with problems independently and adapt to different environments, the brain-inspired function is similar to the human brain, which can effectively make use of visual targets and their surrounding background information to make more efficient and accurate decision results. Currently synthetic aperture radar (SAR) ship target detection has an important role in military and civilian fields, but there are still great challenges in SAR ship target detection due to the problems of large span of ship scales and obvious feature differences. Therefore, this paper proposes an improved anchor-free SAR ship detection algorithm based on brain-inspired attention mechanism, which efficiently focuses on target information ignoring the interference of complex background. First of all, most target detection algorithms are based on the anchor method, which requires a large number of anchors to be defined in advance and has poor generalization capability and performance to be improved in multi-scale ship detection, so this paper adopts an anchor-free detection network to directly enumerate potential target locations to enhance algorithm robustness and improve detection performance. Secondly, in order to improve the SAR ship target feature extraction capability, a dense connection module is proposed for the deep part of the network to promote more adequate deep feature fusion. A visual attention module is proposed for the shallow part of the network to focus on the salient features of the ship target in the local area for the input SAR images and suppress the interference of the surrounding background with similar scattering characteristics. In addition, because the SAR image coherent speckle noise is similar to the edge of the ship target, this paper proposes a novel width height prediction constraint to suppress the noise scattering power effect and improve the SAR ship localization accuracy. Moreover, to prove the effectiveness of this algorithm, experiments are conducted on the SAR ship detection dataset (SSDD) and high resolution SAR images dataset (HRSID). The experimental results show that the proposed algorithm achieves the best detection performance with metrics AP of 68.2% and 62.2% on SSDD and HRSID, respectively.

Uncertainty-aware report generation for chest X-rays by variational topic inference.

Automating report generation for medical imaging promises to minimize labor and aid diagnosis in clinical practice. Deep learning algorithms have recently been shown to be capable of captioning natural photos. However, doing a similar thing for medical data, is difficult due to the variety in reports written by different radiologists with fluctuating levels of knowledge and experience. Current methods for automatic report generation tend to merely copy one of the training samples in the created report. To tackle this issue, we propose variational topic inference, a probabilistic approach for automatic chest X-ray report generation. Specifically, we introduce a probabilistic latent variable model where a latent variable defines a single topic. The topics are inferred in a conditional variational inference framework by aligning vision and language modalities in a latent space, with each topic governing the generation of one sentence in the report. We further adopt a visual attention module that enables the model to attend to different locations in the image while generating the descriptions. We conduct extensive experiments on two benchmarks, namely Indiana U. Chest X-rays and MIMIC-CXR. The results demonstrate that our proposed variational topic inference method can generate reports with novel sentence structure, rather than mere copies of reports used in training, while still achieving comparable performance to state-of-the-art methods in terms of standard language generation criteria.

Deep Active Visual Attention for Real-Time Robot Motion Generation: Emergence of Tool-Body Assimilation and Adaptive Tool-Use

Sufficiently perceiving the environment is a critical factor in robot motion generation. Although the introduction of deep visual processing models have contributed in extending this ability, existing methods lack in the ability to actively modify what to perceive; humans perform internally during visual cognitive processes. This letter addresses the issue by proposing a novel robot motion generation model, inspired by a human cognitive structure. The model incorporates a state-driven active top-down visual attention module, which acquires attentions that can actively change targets based on task states. We term such attentions as role-based attentions, since the acquired attention directed to targets that shared a coherent role throughout the motion. The model was trained on a robot tool-use task, in which the role-based attentions perceived the robot grippers and tool as identical end-effectors, during object picking and object dragging motions respectively. This is analogous to a biological phenomenon called tool-body assimilation, in which one regards a handled tool as an extension of one’s body. The results suggested an improvement of flexibility in model’s visual perception, which sustained stable attention and motion even if it was provided with untrained tools or exposed to experimenter’s distractions.

An Image Captioning Algorithm Based on Combination Attention Mechanism

With the maturity of computer vision and natural language processing technology, we are becoming more ambitious in image captioning. In particular, we are more ambitious in generating longer, richer, and more accurate sentences as image descriptions. Most existing image caption models use an encoder—decoder structure, and most of the best-performing models incorporate attention mechanisms in the encoder—decoder structure. However, existing image captioning methods focus only on visual attention mechanism and not on keywords attention mechanism, thus leading to model-generated sentences that are not rich and accurate enough, and errors in visual feature extraction can directly lead to generated caption sentences that are incorrect. To fill this gap, we propose a combination attention module. This module comprises a visual attention module and a keyword attention module. The visual attention module helps in performing fast extractions of key local features, and the keyword attention module focuses on keywords that may appear in generated sentences. The results generated by the two modules can be corrected for each other. We embed the combination attention module into the framework of the Transformer, thus constructing a new image caption model CAT (Combination Attention Transformer) to generate more accurate and rich image caption sentences. Extensive experiments on the MSCOCO dataset demonstrate the effectiveness and superiority of our method over many state-of-the-art methods.

Emergence of Content-Agnostic Information Processing by a Robot Using Active Inference, Visual Attention, Working Memory, and Planning.

Generalization by learning is an essential cognitive competency for humans. For example, we can manipulate even unfamiliar objects and can generate mental images before enacting a preplan. How is this possible? Our study investigated this problem by revisiting our previous study (Jung, Matsumoto, & Tani, 2019), which examined the problem of vision-based, goal-directed planning by robots performing a task of block stacking. By extending the previous study, our work introduces a large network comprising dynamically interacting submodules, including visual working memory (VWMs), a visual attention module, and an executive network. The executive network predicts motor signals, visual images, and various controls for attention, as well as masking of visual information. The most significant difference from the previous study is that our current model contains an additional VWM. The entire network is trained by using predictive coding and an optimal visuomotor plan to achieve a given goal state is inferred using active inference. Results indicate that our current model performs significantly better than that used in Jung et al. (2019), especially when manipulating blocks with unlearned colors and textures. Simulation results revealed that the observed generalization was achieved because content-agnostic information processing developed through synergistic interaction between the second VWM and other modules during the course of learning, in which memorizing image contents and transforming them are dissociated. This letter verifies this claim by conducting both qualitative and quantitative analysis of simulation results.

THAN: task-driven hierarchical attention network for the diagnosis of mild cognitive impairment and Alzheimer's disease.

To assist doctors to diagnose mild cognitive impairment (MCI) and Alzheimer's disease (AD) early and accurately, convolutional neural networks based on structural magnetic resonance imaging (sMRI) images have been developed and shown excellent performance. However, they are still limited in their capacity in extracting discriminative features because of large sMRI image volumes yet small lesion regions and the small number of sMRI images. We proposed a task-driven hierarchical attention network (THAN) taking advantage of the merits of patch-based and attention-based convolutional neural networks for MCI and AD diagnosis. THAN consists of an information sub-network and a hierarchical attention sub-network. In the information sub-network, an information map extractor, a patch-assistant module, and a mutual-boosting loss function are designed to generate a task-driven information map, which automatically highlights disease-related regions and their importance for final classification. In the hierarchical attention sub-network, a visual attention module and a semantic attention module are devised based on the information map to extract discriminative features for disease diagnosis. Extensive experiments were conducted for four classification tasks: MCI versus (vs.) normal controls (NC), AD vs. NC, AD vs. MCI, and AD vs. MCI vs. NC. Results demonstrated that THAN attained the accuracy of 81.6% for MCI vs. NC, 93.5% for AD vs. NC, 80.8% for AD vs. MCI, and 62.9% for AD vs. MCI vs. NC. It outperformed advanced attention-based and patch-based methods. Moreover, information maps generated by the information sub-network could highlight the potential biomarkers of MCI and AD, such as the hippocampus and ventricles. Furthermore, when the visual and semantic attention modules were combined, the performance of the four tasks was highly improved. The information sub-network can automatically highlight the disease-related regions. The hierarchical attention sub-network can extract discriminative visual and semantic features. Through the two sub-networks, THAN fully exploits the visual and semantic features of disease-related regions and meanwhile considers global features of sMRI images, which finally facilitate the diagnosis of MCI and AD.

Deep Neural Framework with Visual Attention and Global Context for Predicting Image Aesthetics

Computational inference of aesthetics has recently become a hot topic due to its usefulness in widely applications such as evaluating image quality, retouching image and retrieving image. Owing to the subjectivity of this problem, there is no general framework to predict image aesthetics. In this paper, we propose a deep neural framework with visual attention module, self-generated global features and hybrid loss to address this problem. Specifically, the framework can be any state-of-the-art convolution classification network compatible with visual attention. Further, self-generated global feature compensates for the loss of global context information during training stage, and the hybrid loss guides the network to learn the similarity between the predicted aesthetic scores and the ground-truths through fusing soft-max-entropy and Earth Mover’s Distance(EMD). With the above-mentioned improvements, the proposed deep neural framework is capable of effectively predicting image aesthetics in an efficient way. In our experiments, we release a real-world aesthetic dataset that contains 1,800 2K photos labeled by several experienced photographers, and then provide a thorough ablation study of the design choices to better understand the superiority brought by each part of our framework, and design several comparisons with the state-of-the-art methods on a fraction of metrics. The experimental results on two datasets demonstrate that both accuracy and efficiency achieve favorably performance.

Using Action-congruent Language Facilitates the Motor Response during Action Observation: A Combined Transcranial Magnetic Stimulation and Eye-tracking Study.

There is evidence that action observation (AO) and the processing of action-related words are associated with increased activity in cortical motor regions. Research has examined the effects of AO and action verb processing on activity in the motor system independently. The aim of this experiment was to investigate, for the first time, the modulation of corticospinal excitability and visual attention during the concurrent processing of action verbs and AO stimuli. Twenty participants took part in an integrated transcranial magnetic stimulation and eye-tracking protocol. Single-pulse transcranial magnetic stimulation was delivered to the hand representation of the left motor cortex during (i) observation of a static hand, (ii) AO of a hand squeezing a sponge, (iii) AO of the same action with an audio recording of the word "squeeze," and (iv) AO of the same action with an audio recording of the word "green". Motor evoked potentials were recorded from the abductor pollicis brevis and abductor digiti minimi muscles of the right hand. Eye gaze was recorded throughout the four conditions as a proxy for visual attention. Interviews were conducted to discuss participants' preferences and imagery use for each condition. The AO and action verb condition resulted in significantly increased motor evoked potential amplitudes in the abductor pollicis brevis muscle; participants also made significantly more fixations on the sponge and reported wanting to move their hand more in the action verb condition. The inclusion of auditory action verbs, alongside AO stimuli, in movement simulation interventions could have implications for the delivery of AO interventions for motor (re)learning.

Reasoning on the Relation: Enhancing Visual Representation for Visual Question Answering and Cross-Modal Retrieval

Cross-modal analysis has become a promising direction for artificial intelligence. Visual representation is crucial for various cross-modal analysis tasks that require visual content understanding. Visual features which contain semantical information can disentangle the underlying correlation between different modalities, thus benefiting the downstream tasks. In this paper, we propose a Visual Reasoning and Attention Network (VRANet) as a plug-and-play module to capture rich visual semantics and help to enhance the visual representation for improving cross-modal analysis. Our proposed VRANet is built based on the bilinear visual attention module which identifies the critical objects. We propose a novel Visual Relational Reasoning (VRR) module to reason about pair-wise and inner-group visual relationships among objects guided by the textual information. The two modules enhance the visual features at both relation level and object level. We demonstrate the effectiveness of the proposed VRANet by applying it to both Visual Question Answering (VQA) and Cross-Modal Information Retrieval (CMIR) tasks. Extensive experiments conducted on VQA 2.0, CLEVR, CMPlaces, and MS-COCO datasets indicate superior performance comparing with state-of-the-art work.

Cross-modal modulation of cell activity by sound in first-order visual thalamic nucleus.

Cross-modal auditory influence on cell activity in the primary visual cortex emerging at short latencies raises the possibility that the first-order visual thalamic nucleus, which is considered dedicated to unimodal visual processing, could contribute to cross-modal sensory processing, as has been indicated in the auditory and somatosensory systems. To test this hypothesis, the effects of sound stimulation on visual cell activity in the dorsal lateral geniculate nucleus were examined in anesthetized rats, using juxta-cellular recording and labeling techniques. Visual responses evoked by light (white LED) were modulated by sound (noise burst) given simultaneously or 50-400 ms after the light, even though sound stimuli alone did not evoke cell activity. Alterations of visual response were observed in 71% of cells (57/80) with regard to response magnitude, latency, and/or burst spiking. Suppression predominated in response magnitude modulation, but de novo responses were also induced by combined stimulation. Sound affected not only onset responses but also late responses. Late responses were modulated by sound given before or after onset responses. Further, visual responses evoked by the second light stimulation of a double flash with a 150-700 ms interval were also modulated by sound given together with the first light stimulation. In morphological analysis of labeled cells projection cells comparable to X-, Y-, and W-like cells and interneurons were all susceptible to auditory influence. These findings suggest that the first-order visual thalamic nucleus incorporates auditory influence into parallel and complex thalamic visual processing for cross-modal modulation of visual attention and perception.

Effects of subjective visibility on our cognitive function: How visual attentional modulation correlates with subjective visibility

Effects of subjective visibility on our cognitive function: How visual attentional modulation correlates with subjective visibility

Modulation of visual attention assessed using the Stroop task during the menstrual cycle: Comparison between the menstrual and ovulation phases

AbstractWomen might have changes in perception and cognition of colours according to hormonal changes during the menstrual cycle. Colours are often associated with specific emotional facial expressions, but how these two factors interact with each other and differ according to the cycle has remained unclear. We investigated attentional interference between colours and emotional facial expressions for women in the menstrual compared to the ovulation phases by colour judgment based on the Stroop tasks. Participants identified the colours (either red, yellow, or blue) of three types of face stimuli (happy, neutral, or sad) and corresponding scrambled face stimuli by pressing buttons, and we recorded participants' reaction times. The results showed better performance in the menstrual phase than in the ovulation phase. An interference effect was not observed. However, in the scrambled face stimuli task, only the reaction time for happy scrambled face shape and red did not differ between the phases. Thus, participants might pay stronger attention to faces with colours in the menstrual phase than in the ovulation phase but weaker attention to scrambled face shapes in the menstrual phase.

Enhanced Visual Attentional Modulation in Patients with Inherited Peripheral Retinal Degeneration in the Absence of Cortical Degeneration.

The role of attentional mechanisms in peripheral vision loss remains an outstanding question. Our study was aimed at determining the effect of genetically determined peripheral retinal dystrophy caused by Retinitis Pigmentosa (RP) on visual cortical function and tested the recruitment of attentional mechanisms using functional magnetic resonance imaging (fMRI). We included thirteen patients and twenty-two age- and gender-matched controls. We analyzed cortical responses under attentional demands and passive viewing conditions while presenting a visual stimulus covering the central and paracentral visual field. Brain activity was studied in visual areas V1, V2, and V3 as well as in cortical regions of interest corresponding to the preserved and the damaged visual field. The influence of visual field extent and age of disease onset were also investigated. Cortical thickness of visual areas was also measured. We found that cortical visual responses under attentional demands were increased in patients with larger degeneration of visual field, as demonstrated by significant interaction effects between group and task conditions. Moreover, activation during the task condition was increased for patients in two cortical regions of interest corresponding to the preserved and damaged visual field, specifically in patients with severe visual field loss. These findings were observed in the presence of preserved visual cortical structure. We conclude that RP patients have enhanced visual attention recruitment despite their retinal degeneration, while cortical structure and overall response levels remain intact. The unmasking of feedback signals from higher level visual regions involved in attentional processes may explain the increased cortical responses. These findings are relevant for the design of strategies for treating retinal diseases, based on attentional cuing.

Dissociable Catecholaminergic Modulation of Visual Attention: Differential Effects of Catechol-O-Methyltransferase and Dopamine Beta-Hydroxylase Genes on Visual Attention

Visual attention enables us to prioritise behaviourally relevant visual information while ignoring distraction. The neural networks supporting attention are modulated by two catecholamines, dopamine and noradrenaline. The current study investigated the effects of single nucleotide polymorphisms in two catecholaminergic genes – COMT (Val158Met) and DBH (444 G/A) – on individual differences in attention functions. Participants (n = 125) were recruited from the Oxford Biobank by genotype-based recall. They were tested on a continuous performance task (sustained attention), a Go/No-Go task (response inhibition), and a task assessing attentional selection in accordance with the Theory of Visual Attention (TVA). We found a significant effect of DBH genotype status on the capacity to maintain attention over time (sustained attention) as measured by the continuous performance task. Furthermore, we demonstrated a significant association between COMT genotype status and effective threshold of visual perception in attentional selection as estimated based on the TVA task performance. No other group differences in attention function were found with respect to the studied genotypes. Overall, our findings provide novel experimental evidence that: (i) dopaminergic and noradrenergic genotypes have dissociable effects on visual attention; (ii) either insufficient or excessive catecholaminergic activity may have equally detrimental effects on sustained attention.