Attentional Window Research Articles

DSTM: A transformer-based model with dynamic-static feature fusion in speech emotion recognition

With the support of multi-head attention, the Transformer shows remarkable results in speech emotion recognition. However, existing models still suffer from the inability to accurately locate important regions in semantic information at different time scales. To address this problem, we propose a Transformer-based network model for dynamic-static feature fusion, composed of a locally adaptive multi-head attention module and a global static attention module. The locally dynamic multi-head attention module adapts the attention window sizes and window centers of the different regions through speech samples and learnable parameters, enabling the model to adaptively discover and pay attention to valuable information embedded in speech. The global static attention module enables the model to use each element in the sequence fully and learn critical global feature information by establishing connections over the entire input sequence. We also use the data mixture training method to train our model and introduce the CENTER LOSS function to supervise the training of the model, which can better speed up the fitting speed of the model and alleviate the sample imbalance problem to a certain extent. This method achieved good performance on the IEMOCAP and MELD datasets, proving that our proposed model structure and method have better accuracy and robustness.

Visualizing and Understanding Patch Interactions in Vision Transformer.

Vision transformer (ViT) has become a leading tool in various computer vision tasks, owing to its unique self-attention mechanism that learns visual representations explicitly through cross-patch information interactions. Despite having good success, the literature seldom explores the explainability of ViT, and there is no clear picture of how the attention mechanism with respect to the correlation across comprehensive patches will impact the performance and what is the further potential. In this work, we propose a novel explainable visualization approach to analyze and interpret the crucial attention interactions among patches for ViT. Specifically, we first introduce a quantification indicator to measure the impact of patch interaction and verify such quantification on attention window design and indiscriminative patches removal. Then, we exploit the effective responsive field of each patch in ViT and devise a window-free transformer (WinfT) architecture accordingly. Extensive experiments on ImageNet demonstrate that the exquisitely designed quantitative method is shown able to facilitate ViT model learning, leading the top-1 accuracy by 4.28% at most. More remarkably, the results on downstream fine-grained recognition tasks further validate the generalization of our proposal.

SwinShadow: Shifted Window for Ambiguous Adjacent Shadow Detection

Shadow detection is a fundamental and challenging task in many computer vision applications. Intuitively, most shadows come from the occlusion of light by the object itself, resulting in the object and its shadow being contiguous (referred to as the adjacent shadow in this paper). In this case, when the color of the object is similar to that of the shadow, existing methods struggle to achieve accurate detection. To address this problem, we present SwinShadow, a transformer-based architecture that fully utilizes the powerful shifted window mechanism for detecting adjacent shadows. The mechanism operates in two steps. Initially, it applies local self-attention within a single window, enabling the network to focus on local details. Subsequently, it shifts the attention windows to facilitate inter-window attention, enabling the capture of a broader range of adjacent information. These combined steps significantly improve the network's capacity to distinguish shadows from nearby objects. And the whole process can be divided into three parts: encoder, decoder, and feature integration. During encoding, we adopt Swin Transformer to acquire hierarchical features. Then during decoding, for shallow layers, we propose a deep supervision (DS) module to suppress the false positives and boost the representation capability of shadow features for subsequent processing, while for deep layers, we leverage a double attention (DA) module to integrate local and shifted window in one stage to achieve a larger receptive field and enhance the continuity of information. Ultimately, a new multi-level aggregation (MLA) mechanism is applied to fuse the decoded features for mask prediction. Extensive experiments on three shadow detection benchmark datasets, SBU, UCF, and ISTD, demonstrate that our network achieves good performance in terms of balance error rate (BER). The source code and results are now publicly available at https://github.com/harrytea/SwinShadow .

MATNet: multiattention Transformer network for cropland semantic segmentation in remote sensing images

ABSTRACT Remote sensing image semantic segmentation methods have become the main approach for extracting cropland information. However, in the mountainous regions of southwestern China, croplands exhibit narrow and fragmented shapes, as well as complex planting patterns, making it difficult for traditional semantic segmentation methods to accurately delineate fine-grained cropland boundaries. To address these challenges, a multiattention Transformer network named MATNet is proposed in this paper, for fine-grained extraction of cropland at the parcel level in complex scenes. MATNet built upon the fusion of CNN encoder and Transformer decoder. In the encoder, spatial and channel reconstruction units are introduced, reducing information redundancy in the convolutional layers. The Transformer decoder incorporates multiple attention mechanisms, this design feature enhances the attention window's perception of local content and improves the model's ability to extract features from fine-grained cropland parcels through optimized computationnal al location. Taking the experimental results of the Dali cropland dataset as an illustration, MATNet achieved the highest values across five evaluation metrics, including mIoU. Specifically, the Recall, F1, and mIoU scores were 94.68%, 94.69%, and 89.92%, respectively. Compared with six other advanced models, MATNet consistently performed best in terms of extracting fine-grained cropland parcels.

Learning Complementary Spatial-Temporal Transformer for Video Salient Object Detection.

Besides combining appearance and motion information, another crucial factor for video salient object detection (VSOD) is to mine spatial-temporal (ST) knowledge, including complementary long-short temporal cues and global-local spatial context from neighboring frames. However, the existing methods only explored part of them and ignored their complementarity. In this article, we propose a novel complementary ST transformer (CoSTFormer) for VSOD, which has a short-global branch and a long-local branch to aggregate complementary ST contexts. The former integrates the global context from the neighboring two frames using dense pairwise attention, while the latter is designed to fuse long-term temporal information from more consecutive frames with local attention windows. In this way, we decompose the ST context into a short-global part and a long-local part and leverage the powerful transformer to model the context relationship and learn their complementarity. To solve the contradiction between local window attention and object motion, we propose a novel flow-guided window attention (FGWA) mechanism to align the attention windows with object and camera movements. Furthermore, we deploy CoSTFormer on fused appearance and motion features, thus enabling the effective combination of all three VSOD factors. Besides, we present a pseudo video generation method to synthesize sufficient video clips from static images for training ST saliency models. Extensive experiments have verified the effectiveness of our method and illustrated that we achieve new state-of-the-art results on several benchmark datasets.

Intrinsic neural timescales exhibit different lengths in distinct meditation techniques

Meditation encompasses a range of practices employing diverse induction techniques, each characterized by a distinct attentional focus. In Mantra meditation, for instance, practitioners direct their attention narrowly to a given sentence that is recursively repeated, while other forms of meditation such as Shoonya meditation are induced by a wider attentional focus. Here we aimed to identify the neural underpinnings and correlates associated with this spectrum of distinct attentional foci. To accomplish this, we used EEG data to estimate the brain's intrinsic neural timescales (INTs), that is, its temporal windows of activity, by calculating the Autocorrelation Window (ACW) of the EEG signal. The autocorrelation function measures the similarity of a timeseries with a time-lagged version of itself by correlating the signal with itself on different time lags, consequently providing an estimation of INTs length. Therefore, through using the ACW metric, our objective was to explore whether there is a correspondence between the length of the brain's temporal windows of activity and the width of the attentional scope during various meditation techniques. This was performed on three groups of highly proficient practitioners belonging to different meditation traditions, as well as a meditation-naïve control group. Our results indicated that practices with a wider attentional focus, like Shoonya meditation, exhibit longer ACW durations compared to practices requiring a narrower attentional focus, such as Mantra meditation or body-scanning Vipassana meditation. Together, we demonstrated that distinct meditation techniques with varying widths of attentional foci exhibit unique durations in their brain's INTs. This may suggest that the width of the attentional scope during meditation relates and corresponds to the width of the brain's temporal windows in its neural activity. Significance StatementOur research uncovered the neural mechanisms that underpin the attentional foci in various meditation techniques. We revealed that distinct meditation induction techniques, featured by their range of attentional widths, are characterized by varying lengths of intrinsic neural timescales (INTs) within the brain, as measured by the Autocorrelation Window function. This finding may bridge the gap between the width of attentional windows (subjective) and the width of the temporal windows in the brain's neural activity (objective) during different meditation techniques, offering a new understanding of how cognitive and neural processes are related to each other. This work holds significant implications, especially in the context of the increasing use of meditation in mental health and well-being interventions. By elucidating the distinct neural foundations of different meditation techniques, our research aims to pave the way for developing more tailored and effective meditation-based treatments.

ATLAS: Mapping ATtention’s Location And Size to probe five modes of serial and parallel search

Conventional visual search tasks do not address attention directly and their core manipulation of ‘set size’ – the number of displayed items – introduces stimulus confounds that hinder interpretation. However, alternative approaches have not been widely adopted, perhaps reflecting their complexity, assumptions, or indirect attention-sampling. Here, a new procedure, the ATtention Location And Size (‘ATLAS’) task used probe displays to track attention’s location, breadth, and guidance during search. Though most probe displays comprised six items, participants reported only the single item they judged themselves to have perceived most clearly – indexing the attention ‘peak’. By sampling peaks across variable ‘choice sets’, the size and position of the attention window during search was profiled. These indices appeared to distinguish narrow- from broad attention, signalled attention to pairs of items where it arose and tracked evolving attention-guidance over time. ATLAS is designed to discriminate five key search modes: serial-unguided, sequential-guided, unguided attention to ‘clumps’ with local guidance, and broad parallel-attention with or without guidance. This initial investigation used only an example set of highly regular stimuli, but its broader potential should be investigated.

The Effects of High-Intensity Interval Training on Cognitive and Physical Skills in Basketball and Soccer Players.

As athletes pursue excellence, training techniques continue to advance, making structured physical activity an essential tool for enhancing performance. To optimize athletic performance in modern competitive sports, the balance of physical performance and mental clarity is required. This study seeks to examine the effects of High-Intensity Interval Training (HIIT) on cognitive and physical skills in basketball and soccer players. A 3-week HIIT protocol was incorporated based on the Wingate technique. This study included 10 soccer players and 10 basketball players with an average age of 22.79 ± 1.90 years. Participants performed pre- and post-intervention assessments. Physical proficiency was assessed using 20 m sprint, change-of-direction (COD) and dribbling tests, while cognitive skills were assessed using motion object tracking (MOT), working memory, perceptual load (PL), and attention window (AW) tests. The HIIT intervention significantly improved cognitive performance in particular; noteworthy observations were a 15% improvement in motion object tracking test scores and a 16% increase in working memory test scores for basketball players. The attention window test scores showed a 32% increase, and perceptual load test scores were 31% decreased for soccer players post-intervention. There were significant improvements in physical skills; for example, sprint times were decreased by 6%, and change-of-direction and dribbling times were reduced by 8% and 7%, respectively, indicating improved agility, speed, and ball control abilities. In conclusion, both groups performed significantly better on cognitive and physical skill tests post-HIIT intervention.

When temporal attention interacts with expectation

Temporal attention is voluntarily deployed at specific moments, whereas temporal expectation is deployed according to timing probabilities. When the target appears at an expected moment in a sequence, temporal attention improves performance at the attended moments, but the timing and the precision of the attentional window remain unknown. Here we independently and concurrently manipulated temporal attention–via behavioral relevance–and temporal expectation–via session-wise precision and trial-wise hazard rate–to investigate whether and how these mechanisms interact to improve perception. Our results reveal that temporal attention interacts with temporal expectation–the higher the precision, the stronger the attention benefit, but surprisingly this benefit decreased with delayed onset despite the increasing probability of stimulus appearance. When attention was suboptimally deployed to earlier than expected moments, it could not be reoriented to a later time point. These findings provide evidence that temporal attention and temporal expectation are different mechanisms, and highlight their interplay in optimizing visual performance.

SNN4Agents: a framework for developing energy-efficient embodied spiking neural networks for autonomous agents.

Recent trends have shown that autonomous agents, such as Autonomous Ground Vehicles (AGVs), Unmanned Aerial Vehicles (UAVs), and mobile robots, effectively improve human productivity in solving diverse tasks. However, since these agents are typically powered by portable batteries, they require extremely low power/energy consumption to operate in a long lifespan. To solve this challenge, neuromorphic computing has emerged as a promising solution, where bio-inspired Spiking Neural Networks (SNNs) use spikes from event-based cameras or data conversion pre-processing to perform sparse computations efficiently. However, the studies of SNN deployments for autonomous agents are still at an early stage. Hence, the optimization stages for enabling efficient embodied SNN deployments for autonomous agents have not been defined systematically. Toward this, we propose a novel framework called SNN4Agents that consists of a set of optimization techniques for designing energy-efficient embodied SNNs targeting autonomous agent applications. Our SNN4Agents employs weight quantization, timestep reduction, and attention window reduction to jointly improve the energy efficiency, reduce the memory footprint, optimize the processing latency, while maintaining high accuracy. In the evaluation, we investigate use cases of event-based car recognition, and explore the trade-offs among accuracy, latency, memory, and energy consumption. The experimental results show that our proposed framework can maintain high accuracy (i.e., 84.12% accuracy) with 68.75% memory saving, 3.58x speed-up, and 4.03x energy efficiency improvement as compared to the state-of-the-art work for the NCARS dataset. In this manner, our SNN4Agents framework paves the way toward enabling energy-efficient embodied SNN deployments for autonomous agents.

Distinct roles of theta and alpha oscillations in the process of contingent attentional capture.

Visual spatial attention can be captured by a salient color singleton that is contingent on the target feature. A previous study reported that theta (4-7 Hz) and alpha (8-14 Hz) oscillations were related to contingent attentional capture, but the corresponding attentional mechanisms of these oscillations remain unclear. In this study, we analyzed the electroencephalogram data of our previous study to investigate the roles of capture-related theta and alpha oscillation activities. Different from the previous study that used color-changed placeholders as irrelevant cues, the present study adopted abrupt onsets of color singleton cues which tend to elicit phase-locked neural activities. In Experiment 1, participants completed a peripheral visual search task in which spatially uninformative color singleton cues were inside the spatial attentional window and a central rapid serial visual presentation (RSVP) task in which the same cues were outside the spatial attentional window. In Experiment 2, participants completed a color RSVP task and a size RSVP task in which the peripheral color singleton cues were contingent and not contingent on target feature, respectively. In Experiment 1, spatially uninformative color singleton cues elicited lateralized theta activities when they were contingent on target feature, irrespective of whether they were inside or outside the spatial attentional window. In contrast, the same color singleton cues elicited alpha lateralization only when they were inside the spatial attentional window. In Experiment 2, we further found that theta lateralization vanished if the color singleton cues were not contingent on target feature. These results suggest distinct roles of theta and alpha oscillations in the process of contingent attentional capture initiated by abrupt onsets of singleton cues. Theta activities may reflect global enhancement of target feature, while alpha activities may be related to attentional engagement to spatially relevant singleton cues. These lateralized neural oscillations, together with the distractor-elicited N2pc component, might consist of multiple stages of attentional processes during contingent attentional capture.

Diversity of items within attentional window explains “cost-free” diversity judgments

Diversity of items within attentional window explains “cost-free” diversity judgments

Conv-Swinformer: Integration of CNN and shift window attention for Alzheimer’s disease classification

Deep learning (DL) algorithms based on brain MRI images have achieved great success in the prediction of Alzheimer’s disease (AD), with classification accuracy exceeding even that of the most experienced clinical experts. As a novel feature fusion method, Transformer has achieved excellent performance in many computer vision tasks, which also greatly promotes the application of Transformer in medical images. However, when Transformer is used for 3D MRI image feature fusion, existing DL models treat the input local features equally, which is inconsistent with the fact that adjacent voxels have stronger semantic connections than spatially distant voxels. In addition, due to the relatively small size of the dataset for medical images, it is difficult to capture local lesion features in limited iterative training by treating all input features equally. This paper proposes a deep learning model Conv-Swinformer that focuses on extracting and integrating local fine-grained features. Conv-Swinformer consists of a CNN module and a Transformer encoder module. The CNN module summarizes the planar features of the MRI slices, and the Transformer module establishes semantic connections in 3D space for these planar features. By introducing the shift window attention mechanism in the Transformer encoder, the attention is focused on a small spatial area of the MRI image, which effectively reduces unnecessary background semantic information and enables the model to capture local features more accurately. In addition, the layer-by-layer enlarged attention window can further integrate local fine-grained features, thus enhancing the model’s attention ability. Compared with DL algorithms that indiscriminately fuse local features of MRI images, Conv-Swinformer can fine-grained extract local lesion features, thus achieving better classification results.

Attentional asymmetries in peripheral vision.

Previous research on the use of peripheral vision to identify two spatially separated stimuli simultaneously has led to the conclusion that the focus of attention has the form of a symmetric ellipse with a broader expansion along the horizontal compared to the vertical meridian. However, research on pseudoneglect has indicated that attention is not symmetrically distributed to the whole visual field. Here, we test if the attention window is indeed symmetrical with regard to its shape and resolution during peripheral vision. The results indicate that the position of those stimuli relative to the focus of attention influences the ability to identify a given set of stimuli. Specifically, stimuli presented to the left and top of the fixation point were more frequently identified correctly compared to those presented to the right bottom. That is, the attention window is rather not symmetric, which must be considered in future studies on the nature of the focus of attention.

A Critique of the Attentional Window Account of Capture Failures.

There has been a lengthy debate about whether salient stimuli have the power to automatically capture attention, even when entirely task irrelevant. Theeuwes (2022) has suggested that an attentional window account could explain why capture is observed in some studies, but not others. According to this account, when search is difficult, participants narrow their attentional window, and this prevents the salient distractor from generating a saliency signal. In turn, this causes the salient distractor to fail to capture attention. In the present commentary, we describe two major problems with this account. First, the attentional window account proposes that attention must be focused so narrowly that featural information from the salient distractor will be filtered prior to saliency computations. However, many previous studies observing no capture provided evidence that featural processing was sufficiently detailed to guide attention toward the target shape. This indicates that the attentional window was sufficiently broad to allow featural processing. Second, the attentional window account proposes that capture should occur more readily in easy search tasks than difficult search tasks. We review previous studies that violate this basic prediction of the attentional window account. A more parsimonious account of the data is that control over feature processing can be exerted proactively to prevent capture, at least under certain conditions.

The Attentional Capture Debate: When Can We Avoid Salient Distractors and When Not?

There has been a long-standing debate concerning whether we are able to resist attention capture by salient distractors. The so-called "signal suppression hypothesis" of Gaspelin and Luck (2018) claimed to have resolved this debate. According to this view, salient stimuli "naturally attempt to capture attention", yet attention capture may be prevented by a top-down inhibitory mechanism. The current paper describes the conditions in which attention capture by salient distractors can be avoided. Capture by salient items can be avoided when the target is non-salient and therefore difficult to find. Because fine discrimination is needed, a small attentional window is adapted resulting in serial (or partly serial) search. Salient signals outside the focused attentional window do not capture attention anymore not because they are suppressed but because they are ignored. We argue that in studies that have provided evidence for signal suppression, search was likely serial or at least partly serial. When the target is salient, search will be conducted in parallel, and in those cases the salient singleton cannot be ignored nor suppressed but instead will capture attention. We argue that the "signal suppression" account (Gaspelin & Luck, 2018) that seeks to explain resistance to attentional capture has many parallels to classic visual search models such as the "feature integration theory" (Treisman & Gelade, 1980), "feature inhibition" account (Treisman & Sato, 1990), and "guided search" (Wolfe et al, 1989); all models that explain how the serial deployment of attention is guided by the output of earlier parallel processes.

Broader benefits of the pretesting effect: Placement matters.

Taking a test before learning new information in a lesson improves memory for that information - pretesting effect. Although the specific memory benefit of a pretest on pretested information has been well documented, it remains unclear what the circumstances necessary for the broader memory benefit are - that is, the benefit of a pretest on memory of information in the lesson that was not pretested. Sometimes this broader benefit is present, but other times it disappears or reverses. We investigated if manipulating where the non-pretested information appears in a lesson - either before or after the pretested information - affects broader memory benefits. Participants read a text passage (Experiment 1) or watched a video lecture (Experiment 2) after completing a pretest on half of the lesson content. The pretested information appeared either at the beginning (prior to the non-pretested information) or at the end (after the non-pretested information) of the lesson. The final test assessed memory of both pretested and non-pretested information. We hypothesized that pretests trigger an attentional window that opens during the lesson and closes after pretested information has been identified. Any information, including non-pretested information, will benefit from being in this window because it is more likely to be processed. We found that memory of non-pretested information is better if the non-pretested information is presented at the beginning versus at the end of a lesson, regardless of delivery modality. These results indicate that the presentation order of pretested versus non-pretested information contributes to the broader memory benefits associated with pretesting.

Opposing influences of global and local stimulus-hand proximity on crosstalk interference in dual tasks.

In contrast to traditional dualistic views of cognition, visual stimulus processing appears not independent of bodily factors such as hand positioning. For example, reduced crosstalk between two temporally overlapping tasks has been observed when the hands are moved into the attentional window alongside their respective stimuli (i.e., establishing global stimulus-hand proximity). This result indicates that hand-specific attentional processing enhancements support a more serial rather than parallel processing of the two tasks. To further elucidate the nature of these processing modulations and their effect on multitasking performance, the present study consisted of three interrelated crosstalk experiments. Experiment 1 manipulated global stimulus-hand proximity and stimulus-effect proximity orthogonally, with results demonstrating that hand proximity rather than effect proximity drives the crosstalk reduction. Experiment 2 manipulated the physical distance between both hands (i.e., varying local stimulus-hand proximity), with results showing weak evidence of increased crosstalk when both hands are close to each other. Experiment 3 tested opposing influences of global and local stimulus-hand proximity as observed in Experiment 1 and 2 rigorously within one experiment, by employing an orthogonal manipulation of these two proximity measures. Again, we observed slightly increased crosstalk for hands close to each other (replicating Experiment 2); however, in contrast to Experiment 1, the effect of global stimulus-hand proximity on the observed crosstalk was not significant this time. Taken together, the experiments support the notion of hand-specific modulations of perception-action coupling, which can either lead to more or less interference in multitasking, depending on the exact arrangement of hands and stimuli.

Anomaly Detection in Catalog Streams

Detecting anomalies with high accuracy and real time from large amounts of streaming data is a challenge for many real-world applications, such as smart city, astronomical observations, and remote sensing. This article focuses on a special kind of stream, catalog stream, whose high-level catalog structure can be used to analyze the stream effectively. We first formulate the anomaly detection in catalog streams as a constrained optimization problem based on a catalog stream matrix. Then, a novel filtering-identifying based anomaly detection algorithm ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">FIAD</i> ) is proposed, which includes two complementary strategies, true event identifying and false alarm filtering, data-oriented general method and domain-oriented specific method together, to detect truly valuable anomalies. Furthermore, different kinds of attention windows are developed to provide corresponding data for various algorithm components. A scalable and lightweight catalog stream processing framework <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CSPF</i> is designed to support and implement the proposed method efficiently. A prototype system is developed to evaluate the proposed algorithm. Extensive experiments are conducted on the catalog stream data sets from an operational super large field-of-view high-cadence astronomy observation. The experimental results show that the proposed method can achieve a false-positive rate as low as 0.04%, reduces the false alarms by 98.6% compared with the existing methods, and the latency to handle each catalog is 2.1 seconds (much less than the required 15 seconds). Furthermore, a total of 36 transient candidates, including seven microlensing events, 27 superflares, and two dual-superflares, are detected from 21.67 million stars (involving 1.09 million catalogs) from one observation season.

Using Object Reconstruction as a Dynamic Attention Window to Improve Recognition Robustness

Using Object Reconstruction as a Dynamic Attention Window to Improve Recognition Robustness