Top-down Attentional Mechanisms Research Articles

Biologically Inspired Deep Learning Model for Efficient Foveal-Peripheral Vision.

While abundant in biology, foveated vision is nearly absent from computational models and especially deep learning architectures. Despite considerable hardware improvements, training deep neural networks still presents a challenge and constraints complexity of models. Here we propose an end-to-end neural model for foveal-peripheral vision, inspired by retino-cortical mapping in primates and humans. Our model has an efficient sampling technique for compressing the visual signal such that a small portion of the scene is perceived in high resolution while a large field of view is maintained in low resolution. An attention mechanism for performing “eye-movements” assists the agent in collecting detailed information incrementally from the observed scene. Our model achieves comparable results to a similar neural architecture trained on full-resolution data for image classification and outperforms it at video classification tasks. At the same time, because of the smaller size of its input, it can reduce computational effort tenfold and uses several times less memory. Moreover, we present an easy to implement bottom-up and top-down attention mechanism which relies on task-relevant features and is therefore a convenient byproduct of the main architecture. Apart from its computational efficiency, the presented work provides means for exploring active vision for agent training in simulated environments and anthropomorphic robotics.

Multiscale face recognition in cluttered backgrounds based on visual attention

The human vision system can efficiently recognize multiscale objects in cluttered backgrounds. The scheme can be achieved with a visual attention mechanism by concentrating visual resources to the saliency area while ignoring other task-irrelevant areas. However, in the computer vision community, when recognizing multiscale faces in cluttered backgrounds, object detection modules are necessary to locate face regions and reduce the influence of complex backgrounds on the recognition model, which inevitably increases the computational complexity. Motivated by the human vision system, this study proposes the attention developmental network to recognize multiscale faces without using face detectors. A top-down attention mechanism is used to teach the network to focus on the face areas and ignore the backgrounds. An attention-based synapse maintenance mechanism is also introduced to further suppress the background pixels and improve the accuracy of face recognition. Comparative experiments show that our method can attain at least 13% of accuracy improvement over bionic neural networks and ResNet-based recognition networks on the same model scale with less training epochs.

MAResNet: predicting transcription factor binding sites by combining multi-scale bottom-up and top-down attention and residual network.

Accurate identification of transcription factor binding sites is of great significance in understanding gene expression, biological development and drug design. Although a variety of methods based on deep-learning models and large-scale data have been developed to predict transcription factor binding sites in DNA sequences, there is room for further improvement in prediction performance. In addition, effective interpretation of deep-learning models is greatly desirable. Here we present MAResNet, a new deep-learning method, for predicting transcription factor binding sites on 690 ChIP-seq datasets. More specifically, MAResNet combines the bottom-up and top-down attention mechanisms and a state-of-the-art feed-forward network (ResNet), which is constructed by stacking attention modules that generate attention-aware features. In particular, the multi-scale attention mechanism is utilized at the first stage to extract rich and representative sequence features. We further discuss the attention-aware features learned from different attention modules in accordance with the changes as the layers go deeper. The features learned by MAResNet are also visualized through the TMAP tool to illustrate that the method can extract the unique characteristics of transcription factor binding sites. The performance of MAResNet is extensively tested on 690 test subsets with an average AUC of 0.927, which is higher than that of the current state-of-the-art methods. Overall, this study provides a new and useful framework for the prediction of transcription factor binding sites by combining the funnel attention modules with the residual network.

Unsupervised foveal vision neural architecture with top-down attention

Deep learning architectures are an extremely powerful tool for recognizing and classifying images. However, they require supervised learning and normally work on vectors of the size of image pixels and produce the best results when trained on millions of object images. To help mitigate these issues, we propose an end-to-end architecture that fuses bottom-up saliency and top-down attention with an object recognition module to focus on relevant data and learn important features that can later be fine-tuned for a specific task, employing only unsupervised learning. In addition, by utilizing a virtual fovea that focuses on relevant portions of the data, the training speed can be greatly improved. We test the performance of the proposed Gamma saliency technique on the Toronto and CAT 2000 databases, and the foveated vision in the large Street View House Numbers (SVHN) database. The results with foveated vision show that Gamma saliency performs at the same level as the best alternative algorithms while being computationally faster. The results in SVHN show that our unsupervised cognitive architecture is comparable to fully supervised methods and that saliency also improves CNN performance if desired. Finally, we develop and test a top-down attention mechanism based on the Gamma saliency applied to the top layer of CNNs to facilitate scene understanding in multi-object cluttered images. We show that the extra information from top-down saliency is capable of speeding up the extraction of digits in the cluttered multidigit MNIST data set, corroborating the important role of top down attention.

Anxiety and Attentional Processes: The Role of Resting Heart Rate Variability

Individuals with high anxiety preferentially focus attention on emotional information. The autonomic nervous system (ANS) plays an important role in modulating both anxiety and attentional processes. Despite many studies having evaluated attentional bias in anxious people, few of them have investigated the change blindness phenomenon associated with the attentional response toward salient stimuli, considering the role of the ANS. This study aimed to examine the role of heart rate variability (HRV) in trait anxiety and top-down and bottom-up attentional processes toward emotional stimuli. Seventy-five healthy university students were divided into high (N = 39) and low (N = 36) trait anxiety groups and completed a change detection flicker task with neutral, positive, and negative stimuli. The results evidenced a different attentional pattern between people with high and low anxiety considering both the two attentional processes and the valence of the stimuli. Specifically, individuals with high anxiety showed a bias in elaborating emotional stimuli related to their salience (i.e., negative stimuli were faster elaborated than neutral and positive stimuli when top-down attentional mechanisms were involved, while slower performances were highlighted considering bottom-up attentional mechanisms in response to emotional stimuli compared to neutral stimuli). Moreover, an association between HRV, trait anxiety levels, and change blindness phenomenon was confirmed. These results underline the role of HRV as a possible predictor of the alteration of attentional mechanism in anxiety.

A bottom-up and top-down human visual attention approach for hyperspectral anomaly detection

Hyperspectral anomaly detection (HAD) is a branch of target detection which tries to locate pixels that are spectrally or spatially different from their background. In this paper, a visual attention approach is developed to leverage HAD. Traditional HAD methods often try to locate anomalous pixels based on spectral information. However, the spatial features of hyperspectral datasets provide valuable information. Here, we aim to fuse spatial and spectral anomaly features based on bottom-up (BU) and top-down (TD) visual attention mechanisms. Owe to the BU attention, spatial features are extracted by mimicking the primary visual cortex neurons functionality. Also, spectral information is obtained throughout a deep neural network that imitating the TD visual attention. The BU and TD approaches’ results are then integrated to provide both spectral and spatial information. The key findings of our results demonstrate the proposed method outperforms the six state-of-the-art AD methods based on different evaluation metrics.

Causal modulation of right hemisphere\xa0fronto-parietal phase synchrony with Transcranial Magnetic Stimulation during a conscious visual detection task

Correlational evidence in non-human primates has reported increases of fronto-parietal high-beta (22–30 Hz) synchrony during the top-down allocation of visuo-spatial attention. But may inter-regional synchronization at this specific frequency band provide a causal mechanism by which top-down attentional processes facilitate conscious visual perception? To address this question, we analyzed electroencephalographic (EEG) signals from a group of healthy participants who performed a conscious visual detection task while we delivered brief (4 pulses) rhythmic (30 Hz) or random bursts of Transcranial Magnetic Stimulation (TMS) to the right Frontal Eye Field (FEF) prior to the onset of a lateralized target. We report increases of inter-regional synchronization in the high-beta band (25–35 Hz) between the electrode closest to the stimulated region (the right FEF) and right parietal EEG leads, and increases of local inter-trial coherence within the same frequency band over bilateral parietal EEG contacts, both driven by rhythmic but not random TMS patterns. Such increases were accompained by improvements of conscious visual sensitivity for left visual targets in the rhythmic but not the random TMS condition. These outcomes suggest that high-beta inter-regional synchrony can be modulated non-invasively and that high-beta oscillatory activity across the right dorsal fronto-parietal network may contribute to the facilitation of conscious visual perception. Our work supports future applications of non-invasive brain stimulation to restore impaired visually-guided behaviors by operating on top-down attentional modulatory mechanisms.

Goal driven network pruning for object recognition

Pruning studies up to date focused on uncovering a smaller network by removing redundant units, and fine-tuning to compensate accuracy drop as a result. In this study, unlike the others, we propose an approach to uncover a smaller network that is competent only in a specific task, similar to top-down attention mechanism in human visual system. This approach doesn’t require fine-tuning and is proposed as a fast and effective alternative of training from scratch when the network focuses on a specific task in the dataset. Pruning starts from the output and proceeds towards the input by computing neuron importance scores in each layer and propagating them to the preceding layer. In the meantime, neurons determined as worthless are pruned. We applied our approach on three benchmark datasets: MNIST, CIFAR-10 and ImageNet. The results demonstrate that the proposed pruning method typically reduces computational units and storage without harming accuracy significantly.

The ‘Narcissus Effect’: Top-down alpha-beta band modulation of face-related brain areas during self-face processing

Self-related information, such as one’s own face, is prioritized by our cognitive system. Whilst recent theoretical developments suggest that this is achieved by an interplay between bottom-up and top-down attentional mechanisms, their underlying neural dynamics are still poorly understood. Furthermore, it is still matter of discussion as to whether these attentional mechanisms are truly self-specific or instead driven by face familiarity. To address these questions, we used EEG to record the brain activity of twenty-five healthy participants whilst identifying their own face, a friend’s face and a stranger’s face. Time-frequency analysis revealed a greater sustained power decrease in the alpha and beta frequency bands for the self-face, which emerged at late latencies and was maintained even when the face was no longer present. Critically, source analysis showed that this activity was generated in key brain regions for self-face recognition, such as the fusiform gyrus. As in the Myth of Narcissus, our results indicate that one’s own face might have the potential to hijack attention. We suggest that this effect is specific to the self and driven by a top-down attentional control mechanism, which might facilitate further processing of personally relevant events.

Gait Recognition and Understanding Based on Hierarchical Temporal Memory Using 3D Gait Semantic Folding.

Gait recognition and understanding systems have shown a wide-ranging application prospect. However, their use of unstructured data from image and video has affected their performance, e.g., they are easily influenced by multi-views, occlusion, clothes, and object carrying conditions. This paper addresses these problems using a realistic 3-dimensional (3D) human structural data and sequential pattern learning framework with top-down attention modulating mechanism based on Hierarchical Temporal Memory (HTM). First, an accurate 2-dimensional (2D) to 3D human body pose and shape semantic parameters estimation method is proposed, which exploits the advantages of an instance-level body parsing model and a virtual dressing method. Second, by using gait semantic folding, the estimated body parameters are encoded using a sparse 2D matrix to construct the structural gait semantic image. In order to achieve time-based gait recognition, an HTM Network is constructed to obtain the sequence-level gait sparse distribution representations (SL-GSDRs). A top-down attention mechanism is introduced to deal with various conditions including multi-views by refining the SL-GSDRs, according to prior knowledge. The proposed gait learning model not only aids gait recognition tasks to overcome the difficulties in real application scenarios but also provides the structured gait semantic images for visual cognition. Experimental analyses on CMU MoBo, CASIA B, TUM-IITKGP, and KY4D datasets show a significant performance gain in terms of accuracy and robustness.

Simultaneously Detecting and Counting Dense Vehicles From Drone Images

Unmanned aerial vehicles are an essential component in the realization of Industry 4.0. With drones helping to improve industrial safety and efficiency in utilities, construction, and communication, there is an urgent need for drone-based intelligent applications. In this paper, we develop a unified framework to simultaneously detect and count vehicles from drone images. We first explore why the state-of-the-art detectors fail in highly dense drone scenes, which provides more appropriate insights. Then, we propose an effective loss to push the anchors toward matching the ground-truth boxes as much as possible, specifically designed for scale-adaptive anchor generation. Inspired by attention mechanisms in the human visual system, we maximize the mutual information between object classes and features by combining bottom-up cues with top-down attention mechanisms specifically designed for feature extraction. Finally, we build a counting layer with a regularized constraint related to the number of vehicles. Extensive experiments demonstrate the effectiveness of our approach. For both tasks, our proposed method achieves state-of-the-art results on all four challenging datasets. In particular, our results reduce error by a larger factor than previous methods.

Ternary meter from spatial sounds: Differences in neural entrainment between musicians and non-musicians

The present study explores the relationship between the rhythmic structure of music and the spatial dimension of sound. We study how the brain interacts with spatially-separated sounds to build up a metrical structure. Participants listened to sequences of isochronous sounds that came from different positions on the azimuth plane: 0° (control condition), ±30°, ±60° or ±90° (spatial conditions). Ternary meter was signaled by the alternation of one sound on one side and two sounds on the symmetrical side. In Experiment 1, musicians and non-musicians paid attention to the spatial sounds. In Experiment 2, participants paid attention to a visual distractor. We recorded their electroencephalograms and performed frequency-tagging analyses. In both experiments, the isochronous beat elicited steady-state evoked-potentials at the frequency of the beat (2.4 Hz). While in Experiment 1 the alternation produced clear responses at the frequency of the ternary meter (0.8 Hz), in Experiment 2 these responses were only significant in the Spatial 90° condition, and mainly in musicians. This suggests that top-down attentional mechanisms are in play for meter induction. Besides, musicians showed stronger responses to beat and meter than non-musicians, suggesting that formal musical training enhances the neural entrainment to spatially-defined rhythms.

Dilated residual attention network for load disaggregation

Load disaggregation technology is a key technology to realize real-time nonintrusive load monitoring (NILM), and deep learning method has shown great promise for NILM. However, current load disaggregation models based on deep learning are prone to the problems of gradient disappearance and model degradation, and it is difficult to extract effective features from load time series. In order to solve these problems, a new dilated residual attention deep network is proposed for load disaggregation. The proposed model adopts residual learning to extract high-level load features, reduces the difficulty of network optimization and solves the problem of network gradient disappearance. Dilated convolution is introduced to increase the receptive field of convolution kernels, which solves the problem that long-load time-series data are difficult to be learned. Most important of all, the proposed bottom-up and top-down attention mechanism can effectively extract the features of the abrupt points in mains power, improve the accuracy of judging the on/off state of electrical appliances and at the same time improve the learning ability of electrical appliances with low usage. Experiments on WikiEnergy dataset and UK-DALE dataset show that the proposed method achieves more accurate load disaggregation tasks than existing studies, which is of great significance for realizing practical NILM.

A computationally bio-inspired framework of brain activities based on cognitive processes for estimating the depth of anesthesia.

This paper develops a computationally bio-inspired framework of brain activities based on concepts, such as sensory register (SR), encoding, emotion, short-term memory (STM), selective attention, working memory (WM), forgetting, long-term memory (LTM), sustained memory (SM), and response selection for estimating the depth of anesthesia (DOA) using electroencephalogram (EEG) signals. Different brain regions, such as the thalamus, cortex, neocortex, amygdala, striatum, basal ganglia, cerebellum, and hippocampus, are considered for developing a cognitive architecture and a computationally bio-inspired framework. A clinical study was managed on twenty-two patients corresponding to three anesthetic states, including awake state, moderate anesthesia, and general anesthesia. The proposed approach utilizes a multiple of dynamically reconfigurable neural networks with radial basis function (RBF) and its associated data processing mechanisms. The emotion effect in the model, dynamic RBFs in WM and LTMs, and adjusting the adaptive weights in the last layer are the main innovations of the proposed approach. In the proposed approach, various incoming information is entered into the model. The correct labeling process of EEG signals is performed by qualitative and quantitative analyses of peripheral parameters. Then, an SR is used to accumulate the pre-processed EEG segment for a period of 2.3s. Feature extraction is performed in the encoding stage as a primary perception. The output of this stage can be transferred to STM and WM with a bottom-up involuntary attentional capture. LTM and SM are a fairly permanent reservoir for information which is passed from WM using a top-down voluntary attention mechanism. Finally, weighting factors in SM and LTMs outputs are determined and then response selection is used by winner-take-all (WTA) strategy. The results indicate that the proposed approach can classify in different anesthetic states with an average accuracy of 89.2%. Results also indicate that the combined use of the above elements can effectively decipher the cognitive process task. A final comparison between the obtained results and the previous method on the same database indicate the effectiveness of the proposed approach for estimating DOA.

Functional MRI evidence of the cortico-olivary efferent pathway during active auditory target processing in humans

Auditory target detection has been explored by a number of studies, but none have demonstrated activity in the auditory subcortical centers evoked by the top-down attentional mechanism related to target detection in humans. We applied functional magnetic resonance imaging (fMRI) with sparse sampling to explore activity in the auditory centers, particularly in the subcortex, during an active auditory target detection task. Fourteen healthy subjects with normal hearing tapped the left index finger in response to target tonal stimuli presented among other (non-target) stimuli during continuous white noise stimulation. General linear model, region-of-interest, and connectivity analyses were performed. In the cortex, bilateral auditory cortices as well as the cingulate gyrus, thalamus, and supramarginal gyrus were activated to target stimuli and functionally connected to each other. In the subcortex, the superior olivary complex (SOC) and locus coeruleus were activated to the target but not to the non-target or background noise stimuli. The SOC was the only auditory subcortical center that displayed connectivity to the auditory cortical areas as well as the cingulate and supramarginal gyri during target presentation but not during other conditions. SOC activation appears to be the first fMRI evidence of direct cortico-olivary projections in the human brain as well as SOC participation in auditory target detection. Our results may be an initial step towards developing a noninvasive methodology to evaluate the functional integrity of the auditory efferent system in humans.

VAA: Visual Aligning Attention Model for Remote Sensing Image Captioning

Owing to the effectiveness in selectively focusing on regions of interest of images, the attention mechanism has been widely used in image caption task, which can provide more accurate image information for training deep sequential models. Existing attention-based models typically rely on top-down attention mechanism. While somewhat effective, attention masks in these attention-based models are queried from image features by hidden states of LSTM, rather than optimized by the objective functions. This indirectly supervised training approach cannot ensure that attention layers accurately focus on regions of interest. To address the above issue, in this paper, a novel attention model, Visual Aligning Attention model (VAA), is proposed. In this model, the attention layer is optimized by a well-designed visual aligning loss during the training phase. The visual aligning loss is obtained by explicitly calculating the feature similarity of attended image features and corresponding word embedding vectors. Besides, in order to eliminate the influence of non-visual words in training the attention layer, a visual vocab used for filtering out non-visual words in sentences is proposed, which can neglect the non-visual words when calculating the visual aligning loss. Experiments on UCM-Captions and Sydney-Captions prove that the proposed method is more effective in remote sensing image caption task.

Dissociating top-down and bottom-up temporal attention in Down syndrome: A neurocostructive perspective

Recent studies suggest that both top-down and bottom-up temporal orienting (TO) of attention are established early and show stable trajectories in typical development. Yet, no evidence is available about atypical development of TO. In the present study, we first investigate the interplay between top-down and bottom-up temporal attention mechanisms in Down Syndrome (DS). Children and adolescents with DS (n = 33) and typically developing individuals matched to children with DS by either chronological (TD-CA; n = 33) or mental age (TD-MA; n = 31) performed a simple cued reaction time designed to assess top-down (endogenous TO) and bottom-up (foreperiod and sequential effect) temporal attention. Mean task accuracy was on average lower and more variable in the DS group than controls and it was predicted by both verbal and non-verbal ability. A subsample of “high-performing” children, showing comparable task speed and higher accuracy (≥65%), was then extracted from each group to test the presence of top-down and bottom-up temporal attention while limiting possible floor effects and overly slow responding. While the control groups showed both top-down and bottom-up TO effects, only the latter were present in DS group. Consistent with a neuroconstructivist account, this study identifies an atypical domain-general cognitive mechanism, temporal orienting of attention, that may constrain the build-up of domain-specific skills in DS, and is for investigation in other atypically developing groups.

The power of negative thinking: Paradoxical but effective ignoring of salient-but-irrelevant stimuli with a spatial cue

ABSTRACTA uniquely coloured singleton among other uniformly coloured stimuli can function as a powerful attractor of attention. However, top-down attentional mechanisms are also powerful, and there are circumstances in which they can suppress irrelevant distractors. The current study tested whether an endogenous spatial cue indicating the location of a salient colour singleton distractor can eliminate involuntary attentional allocation to such a stimulus. When an arrow cue indicated a to-be-ignored location that would never contain a target but would contain a colour singleton distractor, a significant singleton capture effect was eliminated on ignore trials regardless of the consistency of the singleton colour in Experiments 1 and 2. In Experiment 3, a consistent colour singleton was used without a spatial cue. Participants were not able to suppress singleton distractors in the absence of a cue. However, in Experiments 1 & 2, the status of a letter inscribed in the singleton (i.e., whether it is compatible or incompatible with the target of the search) did affect reaction time on ignore trials. If the compatibility effect indicates a shift of attention to the singleton, the data may mean that, with a spatial cue, participants inhibit a cued location by first selecting the location and then rapidly disengaging from it.

Functional Diversity of Thalamic Reticular Subnetworks.

The activity of the GABAergic neurons of the thalamic reticular nucleus (TRN) has long been known to play important roles in modulating the flow of information through the thalamus and in generating changes in thalamic activity during transitions from wakefulness to sleep. Recently, technological advances have considerably expanded our understanding of the functional organization of TRN. These have identified an impressive array of functionally distinct subnetworks in TRN that participate in sensory, motor, and/or cognitive processes through their different functional connections with thalamic projection neurons. Accordingly, “first order” projection neurons receive “driver” inputs from subcortical sources and are usually connected to a densely distributed TRN subnetwork composed of multiple elongated neural clusters that are topographically organized and incorporate spatially corresponding electrically connected neurons—first order projection neurons are also connected to TRN subnetworks exhibiting different state-dependent activity profiles. “Higher order” projection neurons receive driver inputs from cortical layer 5 and are mainly connected to a densely distributed TRN subnetwork composed of multiple broad neural clusters that are non-topographically organized and incorporate spatially corresponding electrically connected neurons. And projection neurons receiving “driver-like” inputs from the superior colliculus or basal ganglia are connected to TRN subnetworks composed of either elongated or broad neural clusters. Furthermore, TRN subnetworks that mediate interactions among neurons within groups of thalamic nuclei are connected to all three types of thalamic projection neurons. In addition, several TRN subnetworks mediate various bottom-up, top-down, and internuclear attentional processes: some bottom-up and top-down attentional mechanisms are specifically related to first order projection neurons whereas internuclear attentional mechanisms engage all three types of projection neurons. The TRN subnetworks formed by elongated and broad neural clusters may act as templates to guide the operations of the TRN subnetworks related to attentional processes. In this review article, the evidence revealing the functional TRN subnetworks will be evaluated and will be discussed in relation to the functions of the various sensory and motor thalamic nuclei with which these subnetworks are connected.

The Power of Negative Thinking: Paradoxical but Effective Ignoring of Salient-but-Irrelevant Stimuli by a Spatial Cue

It is often assumed that a uniquely colored singleton among other uniformly colored stimuli can function as a powerful attractor of attention (Theeuwes, 1992). However, top-down attentional mechanisms are also powerful and can suppress distractors when a pre-cue is given (Cunningham & Egeth, 2016; Munneke, Van der Stigchel, & Theeuwes, 2008). The current study tested whether an endogenous spatial cue indicating the location of a salient color singleton distractor can eliminate involuntary attentional allocation to such a stimulus. On each trial, either a capital "B" or "F" was randomly presented as the target letter. On ignore trials, an arrow cue indicated a to-be-ignored location that would never contain a target but would contain a color singleton distractor. Neutral trials provided no useful spatial information. Target-distractor compatibility was manipulated with a lowercase 'b' or 'f' distractor to test processing of the cued location. The results showed that a significant singleton capture effect was eliminated on ignore trials regardless of the consistency of the singleton color (Experiments 1 & 2). Significant compatibility effects were observed on both ignore and neutral trials, suggesting that participants adopted the strategy of selecting the cued location followed by rapid disengagement from that location on ignore trials. Overall, a spatial cue produces paradoxical but effective ignoring of salient-but-irrelevant stimuli; with a spatial cue, participants inhibited a cued location by first selecting the location and then rapidly disengaging from it. To explore whether participants could learn to ignore the color singleton in the absence of a spatial cue (Gaspelin & Luck, 2017; Vatterott & Vecera, 2012), in Experiment 3, only neutral trials were used with a consistent color singleton. Participants were not able to suppress singleton distractors, although the magnitude of singleton capture effects was reduced compared to that of neutral trials in Experiments 1 and 2. Meeting abstract presented at VSS 2018