Top-down Attention Model Research Articles

Coherent Visual Storytelling via Parallel Top-Down Visual and Topic Attention

Visual storytelling aims at producing a narrative paragraph for a given photo album automatically. It introduces more new challenges than individual image paragraph descriptions, mainly due to the difficulty in preserving coherent topics and in generating diverse phrases to depict the rich content of a photo album. Existing attention-based models that lack higher-level guiding information always result in a deviation between the generated sentence and the topic expressed by the image. In addition, these widely applied language generation approaches employing standard beam search tend to produce monotonous descriptions. In this work, a coherent visual storytelling (CoVS) framework is designed to address the above-mentioned problems. Specifically, in the encoding phase, an image sequence encoder is designed to efficiently extract visual features of the input photo album. Then, the novel parallel top-down visual and topic attention (PTDVTA) decoder is constructed via a topic-aware neural network, a parallel top-down attention model, and a coherent language generator. Concretely, visual attention focuses on the attributes and the relationships of the objects, while topic attention integrating a topic-aware neural network could improve the coherence of generated sentences. Eventually, a phrase beam search algorithm with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$n$ </tex-math></inline-formula> -gram hamming diversity is further designed to optimize the expression diversity of the generated story. To justify the proposed CoVS framework, extensive experiments are conducted on the VIST dataset, which shows that CoVS can automatically generate coherent and diverse stories in a more natural way. Moreover, CoVS obtains better performance than state-of-the-art baselines on BLEU-4 and METEOR scores, while maintaining good CIDEr and ROUGH_L scores. The source code of this work can be found in <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://mic.tongji.edu.cn</uri> .

Stack-VS: Stacked Visual-Semantic Attention for Image Caption Generation

Recently, automatic image caption generation has been an important focus of the work on multimodal translation task. Existing approaches can be roughly categorized into two classes, top-down and bottom-up , the former transfers the image information (called as visual-level feature ) directly into a caption, and the later uses the extracted words (called as semantic-level attribute ) to generate a description. However, previous methods either are typically based one-stage decoder or partially utilize part of visual -level or semantic -level information for image caption generation. In this paper, we address the problem and propose an innovative multi-stage architecture (called as Stack-VS ) for rich fine-grained image caption generation, via combining bottom-up and top-down attention models to effectively handle both visual -level and semantic -level information of an input image. Specifically, we also propose a novel well-designed stack decoder model, which is constituted by a sequence of decoder cells, each of which contains two LSTM-layers work interactively to re-optimize attention weights on both visual-level feature vectors and semantic-level attribute embeddings for generating a fine-grained image caption. Extensive experiments on the popular benchmark dataset MSCOCO show the significant improvements on different evaluation metrics, i.e., the improvements on BLEU-4 / CIDEr / SPICE scores are 0.372, 1.226 and 0.216, respectively, as compared to the state-of-the-art.

Attention-based active visual search for mobile robots

We present an active visual search model for finding objects in unknown environments. The proposed algorithm guides the robot towards the sought object using the relevant stimuli provided by the visual sensors. Existing search strategies are either purely reactive or use simplified sensor models that do not exploit all the visual information available. In this paper, we propose a new model that actively extracts visual information via visual attention techniques and, in conjunction with a non-myopic decision-making algorithm, leads the robot to search more relevant areas of the environment. The attention module couples both top-down and bottom-up attention models enabling the robot to search regions with higher importance first. The proposed algorithm is evaluated on a mobile robot platform in a 3D simulated environment. The results indicate that the use of visual attention significantly improves search, but the degree of improvement depends on the nature of the task and the complexity of the environment. In our experiments, we found that performance enhancements of up to 42% in structured and 38% in highly unstructured cluttered environments can be achieved using visual attention mechanisms.

A novel visual attention method for target detection from SAR images

Synthetic Aperture Radar (SAR) imaging systems have been widely used in civil and military fields due to their all-weather and all-day abilities and various other advantages. However, due to image data exponentially increasing, there is a need for novel automatic target detection and recognition technologies. In recent years, the visual attention mechanism in the visual system has helped humans effectively deal with complex visual signals. In particular, biologically inspired top-down attention models have garnered much attention recently. This paper presents a visual attention model for SAR target detection, comprising a bottom-up stage and top-down process. In the bottom-up step, the Itti model is improved based on the difference between SAR and optical images. The top-down step fully utilizes prior information to further detect targets. Extensive detection experiments carried out on the benchmark Moving and Stationary Target Acquisition and Recognition (MSTAR) dataset show that, compared with typical visual models and other popular detection methods, our model has increased ability and robustness for SAR target detection, under a range of Signal to Clutter Ratio (SCR) conditions and scenes. In addition, results obtained using only the bottom-up stage are inferior to those of the proposed method, further demonstrating the effectiveness and rationality of a top-down strategy. In summary, our proposed visual attention method can be considered a potential benchmark resource for the SAR research community.

Vulnerable Plaque Recognition Based on Attention Model with Deep Convolutional Neural Network.

Previous studies have proved that the vulnerable plaque is a major factor leading to the onset of acute coronary syndrome (ACS). Recognizing vulnerable plaques is essential for cardiologists to treat illnesses, early. However, this task often comes with the challenge of insufficient annotated data sets and subtle differences between lesion regions and normal regions. In this paper, we apply the visual attention model with deep neural network to improve the performance of recognizing vulnerable plaques. There are two key ideas about our method: 1) using a top-down attention model to extract salient regions (blood vessels) according to the doctor's prior knowledge, and 2) employing a multi-task neural network to complete the recognition task. The first branch, a typical classification task, is to distinguish whether the image contains vulnerable plaques. The other branch uses a column-wise segmentation to locate vulnerable plaques. We have verified the effectiveness of our proposed method on the data set provided by 2017 CCCV-IVOCT Challenge. The proposed method obtains good performance.

Fabric defect detection using an integrated model of bottom-up and top-down visual attention

In this paper, we introduce visual attention mechanism from the human vision system to detect fabric defects, and propose the integrated computational model of top-down and bottom-up visual attention. Firstly, utilizing data driven of bottom-up visual attention generates the overall saliency map to pop out fabric defects. Secondly, using target feature driven (task driven) of top-down visual attention forms region of interest (ROI) of fabric defects. Finally, the fabric defects are segmented from ROI using the threshold. Experimental results show that compared with the traditional detection methods, the proposed algorithm can segment accurately common defects from fabric images and enhance detection rate, and it has strong universality for different fabric texture, which can provide the possibility for the realization of automatic fabric defect detection.

Priming of awareness or how not to measure visual awareness

A foundational issue in the study of unconscious processing concerns whether the stimuli of interest are truly out of awareness. Objective methods employing forced choice are typically championed as the gold standard and widely thought to be conservative. Here, however, as a case study, we demonstrate an underestimation of awareness in a collection of studies on unconscious cognitive control. Specifically, we found that (a) in addition to genuine unawareness, chance performance could be due to a failure to perform the task; (b) visual awareness for low-visibility trials was elevated when mixed with high-visibility trials compared with when presented alone as demonstrated in both objective awareness (forced-choice performance) and subjective awareness (rating based on a perceptual awareness scale); and (c) the elevation effect was partly due to a shape-specific template enhancement at both the block and intertrial levels. We term the awareness elevation effect priming of awareness: Visual priming fundamentally alters awareness, boosting otherwise invisible objects into consciousness. These results implicate two key requirements for measuring awareness: (a) verify that participants are truly performing the awareness task and (b) use all types of trials in the awareness test as in the main experiment. Priming of awareness is consistent with an expanded model of awareness and top-down attention in which awareness is determined by (a) retinal stimulus strength and (b) both goal-dependent and goal-independent extra-retinal modulation.

A Top-down Attention Model Using Naive Bayesian Network

본 논문에서는 탐색하고자 하는 물체의 정보가 되는 특징을 추출하고, 이를 나이브 베이지안 네트워크로 학습하여 물체 탐색 시 바이어싱 정보로 사용하는 하향식 시각주의 모델을 제안한다. 기존의 상향식 시각주의 모델에서는 탐색하고자 하는 물체에 대한 선지식을 사용하지 않기 때문에 탐색하고자 하는 물체가 현저도가 낮을 경우 미탐색되는 한계가 있었다. 본 논문에서 제안하는 모델에서는 학습을 통해 고차원적인 정보를 특징정보로 사용함으로써 상향식 모델에서는 현저도가 낮게 결정되어 미탐색되었던 물체를 탐색할 수 있도록 하였다. 성능 평가를 위하여 특정 물체 탐색 문제에 적용하였으며 기존의 상향식 모델 및 하향식 모델과 비교하였다.

Top-down attention based on object representation and incremental memory for knowledge building and inference

Humans can efficiently perceive arbitrary visual objects based on an incremental learning mechanism with selective attention. This paper proposes a new task specific top-down attention model to locate a target object based on its form and color representation along with a bottom-up saliency based on relativity of primitive visual features and some memory modules. In the proposed model top-down bias signals corresponding to the target form and color features are generated, which draw the preferential attention to the desired object by the proposed selective attention model in concomitance with the bottom-up saliency process. The object form and color representation and memory modules have an incremental learning mechanism together with a proper object feature representation scheme. The proposed model includes a Growing Fuzzy Topology Adaptive Resonance Theory (GFTART) network which plays two important roles in object color and form biased attention; one is to incrementally learn and memorize color and form features of various objects, and the other is to generate a top-down bias signal to localize a target object by focusing on the candidate local areas. Moreover, the GFTART network can be utilized for knowledge inference which enables the perception of new unknown objects on the basis of the object form and color features stored in the memory during training. Experimental results show that the proposed model is successful in focusing on the specified target objects, in addition to the incremental representation and memorization of various objects in natural scenes. In addition, the proposed model properly infers new unknown objects based on the form and color features of previously trained objects.

The co-attention model for tiny activity analysis

Automatic understanding of human activities is a huge challenge in multimedia analysis field. This challenge is especially critical in small-scale activities, such as finger motions, and activities in complex scenes. For typical camera views, both global feature and local feature analysis methods are unsuitable. To solve this problem, many studies focus on using spatio-temporal features and feature selection methods to get video representation. However, these spatio-temporal features are problematic for two reasons. First, we are not sure whether these features are meaningful foreground or noise. Second, we are unable to foresee where an activity will occur based on these features. Therefore, a biological feature selection method is needed to reorganize these spatio-temporal features and represent the video in a feature space. In this paper, we propose a graph based Co-Attention model to select more efficient features for activity analysis. Without reducing the dimensionality, our Co-Attention model considers the number of interest points. Our model is derived from correlations among individual tiny activities, whose salient regions are identified by combining an integrated top-down and bottom-up visual attention model, and a motion attention model built by spatio-temporal features instead of optical flow directly. Different from typical attention models, the Co-Attention model allows multiple regions of interest in video co-existing for further analysis. Experimental results on the KTH dataset, YouTube dataset and a new tiny activity dataset, Pump dataset which consist of visual observation data from patients operating an infusion pump, validate our activity analysis approach is more effective than state-of-the-art methods.

Computational model of excitatory/inhibitory ratio imbalance role in attention deficit disorders

Impairments in attentional behaviors, including over-selectivity, under-selectivity, distractibility and difficulty in shift of attention, are widely reported in several developmental disorders, including autism. Uncharacteristic inhibitory to excitatory neuronal number ratio (IER) and abnormal synaptic strength levels in the brain are two broadly accepted neurobiological disorders observed in autistic individuals. These neurobiological findings are contrasting and their relation to the atypical attentional behaviors is not clear yet. In this paper, we take a computational approach to investigate the relation of imbalanced IER and abnormal synaptic strength to some well-documented spectrum of attentional impairments. The computational model is based on a modified version of a biologically plausible neural model of two competing minicolumns in IT cortex augmented with a simple model of top-down attention. Top-down attention is assumed to amplify (attenuates) attended (unattended) stimulus. The inhibitory synaptic strength parameter in the model is set such that typical attentional behavior is emerged. Then, according to related findings, the parameter is changed and the model's attentional behavior is considered. The simulation results show that, without any change in top-down attention, the abnormal inhibitory synaptic strength values--and IER imbalance- result in over-selectivity, under-selectivity, distractibility and difficulty in shift of attention in the model. It suggests that the modeled neurobiological abnormalities can be accounted for the attentional deficits. In addition, the atypical attentional behaviors do not necessarily point to impairments in top-down attention. Our simulations suggest that limited changes in the inhibitory synaptic strength and variations in top-down attention signal affect the model's attentional behaviors in the same way. So, limited deficits in the inhibitory strength may be alleviated by appropriate change in top-down attention biasing. Nevertheless, our model proposes that this compensation is not possible for very high and very low values of the inhibitory strength.

Selective attention-based novelty scene detection in dynamic environments

We propose a biologically motivated novelty detection model of a scene that can give a robust performance for natural color scenes with an affine transformed field of view, as well as noisy scenes in a dynamic visual environment. Novelty detection is an essential property for developmental robots. A topology of a visual scan path of an input scene and an energy signature for the corresponding visual scan path are obtained and considered when deciding on a novelty occurrence in an input scene. The visual scan path is generated by a low-level top-down attention model in conjunction with a bottom-up saliency map model.

MODELING AUDITORY PATHWAY FOR INTELLIGENT INFORMATION ACQUISITION

An auditory model has been developed for an intelligent speech information acquisition system in real-world noisy environment. The developed mathematical model of the human auditory pathway consists of three components, i.e. the nonlinear feature extraction from cochlea to auditory cortex, the binaural processing at superior olivery complex, and the top-down attention from higher brain to the cochlea. The feature extraction is based on information-theoretic sparse coding throughout the auditory pathway. Also, the time-frequency masking is incorporated as a model of the lateral inhibition in both time and frequency domain. The binaural processing is modeled as the blind signal separation and adaptive noise canceling based on the independent component analysis with hundreds of time-delays for noisy reverberated signals. The Top-Down (TD) attention comes from familiarity and/or importance of the sensory information, i.e. the sound, and a simple but efficient TD attention model had been developed based on the error backpropagation algorithm. Also, the binaural processing and top-down attention are combined for speech signals with heavy noises. This auditory model requires extensive computing, and special hardware had been developed for real-time applications. Experimental results demonstrate much better recognition performance in real-world noisy environments.