Spatial Attention Network Research Articles

Spatial and task attention network for treatment response prediction in locally advanced cervical cancer radiotherapy

Cervical cancer is a common gynecological tumor treated predominantly with radiotherapy for locally advanced cases. However, despite treatment, almost one-third of patients experience recurrence within 18 months. Accurate prediction of patient response to therapy is critical for selecting optimal treatment plans. Currently, MRI images are manually segmented to identify the tumor region and predict treatment response using image information within the tumor. However, current methods treat segmentation and response prediction as separate tasks and manual segmentation can be expensive. To address these issues, we propose a spatial and task attention network that simultaneously segments the tumor and predicts the response to cervical cancer radiotherapy. Our approach employs a spatial attention module to focus on the tumor region and a task attention module to explore the correlation between tumor segmentation and treatment response prediction, achieving automatic segmentation of the tumor. We retrospectively collected MRI images from 138 patients with locally advanced cervical cancer before radiotherapy and conducted 5-fold cross-validation experiments, demonstrating that our method achieves competitive results.

Point-wise spatial network for identifying carcinoma at the upper digestive and respiratory tract

ProblemArtificial intelligence has been widely investigated for diagnosis and treatment strategy design, with some models proposed for detecting oral pharyngeal, nasopharyngeal, or laryngeal carcinoma. However, no comprehensive model has been established for these regions.AimOur hypothesis was that a common pattern in the cancerous appearance of these regions could be recognized and integrated into a single model, thus improving the efficacy of deep learning models.MethodsWe utilized a point-wise spatial attention network model to perform semantic segmentation in these regions.ResultsOur study demonstrated an excellent outcome, with an average mIoU of 86.3%, and an average pixel accuracy of 96.3%.ConclusionThe research confirmed that the mucosa of oral pharyngeal, nasopharyngeal, and laryngeal regions may share a common appearance, including the appearance of tumors, which can be recognized by a single artificial intelligence model. Therefore, a deep learning model could be constructed to effectively recognize these tumors.

SSANet: spatial stain attention network for pathological images classification

SSANet: spatial stain attention network for pathological images classification

A macro–micro spatio-temporal neural network for traffic prediction

Accurate traffic prediction is crucial for planning, management and control of intelligent transportation systems. Most state-of-the-art methods for traffic prediction effectively capture complex traffic patterns (e.g. spatial and temporal correlations of traffic data) by employing spatio-temporal neural networks as prediction models, together with graph convolution networks to learn spatial correlations of prediction objects (e.g. traffic states of road segments, as in this study). Such spatial correlations can be regarded as micro correlations. However, there are also macro correlations between regions, each of which is composed of multiple road segments or artificially partitioned areas. Macro correlations represent another type of interaction within road segments, and should be carefully considered when predicting traffic. The diversity of micro spatial correlations and corresponding macro spatial correlations (e.g. correlations based on physical proximity or traffic pattern similarity) further increases the complexity of traffic prediction. We overcome these challenges by developing a macro–micro spatio-temporal neural network model, denoted ‘MMSTNet’. MMSTNet captures spatio-temporal patterns by (a) utilizing a graph convolution network and a spatial attention network to capture micro and macro spatial correlations, respectively; (b) employing a temporal convolution network and a temporal attention network to learn temporal patterns; and (c) integrating hierarchically learned representations based on designed attention mechanisms. We perform evaluations on two real-world datasets and thereby demonstrate that MMSTNet outperforms state-of-the-art models in traffic prediction tasks.

A modulized lane‐follower for driverless vehicles using multi‐frame

AbstractAs a fundamental function, lane following plays an important role for driverless vehicles. Unfortunately, lane followers generally confront great difficulty in lane line missed situations caused by vague line, shadows etc. However, for most lane line missed situation, clues of the line may be hidden in prior view of it. Consequently, a lane follower called UNL Lane Follower, which contains two deep learning network modules is proposed. The first module is a lane line detection model called UNET_CLB. Here, the sequence of image frames is utilised rather than only the current frame to deal with the missing lane lines. The second module is a lane‐following model called LSTM_DTS, which combines a deep learning attention mechanism (temporal attention network and spatial attention network) with a recurrent neural network. As a result, the proposed UNL Lane Follower produces smoother driving behaviour, especially when a lane line is temporally missed. For better explain ability, the role of each part of the network structure is analysed and explained intuitively. As a modularised network, the UNET_CLB is firstly trained and tested on the TuSimple dataset and CULane dataset. The LSTM_DTS lane follow is then trained and tested on our actual lane following dataset. Finally, the UNL Lane Follower is trained and tested as a whole in a simulation running on Webots, after importing the weight of the two modules trained separately. All testing results showed that the UNL Lane Follower can provide better robustness and accuracy for lane line following mission in the line missed situations.

MSCSA-Net: Multi-Scale Channel Spatial Attention Network for Semantic Segmentation of Remote Sensing Images

Although deep learning-based methods for semantic segmentation have achieved prominent performance in the general image domain, semantic segmentation for high-resolution remote sensing images remains highly challenging. One challenge is the large image size. High-resolution remote sensing images can have very high spatial resolution, resulting in images with hundreds of millions of pixels. This makes it difficult for deep learning models to process the images efficiently, as they typically require large amounts of memory and computational resources. Another challenge is the complexity of the objects and scenes in the images. High-resolution remote sensing images often contain a wide variety of objects, such as buildings, roads, trees, and water bodies, with complex shapes and textures. This requires deep learning models to be able to capture a wide range of features and patterns to segment the objects accurately. Moreover, remote sensing images can suffer from various types of noise and distortions, such as atmospheric effects, shadows, and sensor noises, which can also increase difficulty in segmentation tasks. To deal with the aforementioned challenges, we propose a new, mixed deep learning model for semantic segmentation on high-resolution remote sensing images. Our proposed model adopts our newly designed local channel spatial attention, multi-scale attention, and 16-piece local channel spatial attention to effectively extract informative multi-scale features and improve object boundary discrimination. Experimental results with two public benchmark datasets show that our model can indeed improve overall accuracy and compete with several state-of-the-art methods.

DenseGCN: A multi‐level and multi‐temporal graph convolutional network for action recognition

AbstractWith the exponential growth of video data, action recognition has become an increasingly important area of study. Despite various advancements, achieving a balance between detection accuracy and lightness remains a formidable challenge, primarily due to the complexity of existing action recognition models. To address this issue, DenseGCN is developed, a lightweight network designed to optimize accuracy and efficiency. The aim was to create a detection model that has high accuracy while remaining lightweight for real‐world applications. DenseGCN operates via a unique three‐level feature fusion system. The initial stage involves the Multi‐level Fusion Network (MlFN), which contains dense connections and a Spatial‐Temporal Fusion Attention module (STF‐Att), designed to eliminate bias in feature extraction caused by deep networks. In the next stage, RefineBone tackles optimization issues in low‐dimensional feature layers by leveraging high‐dimensional feature layers, thus avoiding gradient stacking. Finally, the Multi‐temporal Fusion Feature Pyramid Network (MF‐FPN) generates a discriminative classification feature map by repetitively combining data from multiple dimensions. This strategy has proven successful in refining the extracted feature, allowing for discriminative feature extraction even with a reduced number of channels. This efficient design not only contributes to further research in developing lightweight networks but also offers enhanced possibilities for real‐world implementations. In two large‐scale datasets, NTU RGB+D 60 and 120, DenseGCN outperformed other state‐of‐the‐art methods, achieving an accuracy of 92.7% on the X‐View benchmark of the NTU RGB+D 60 dataset. The DenseGCN is 10.2 × faster and 10 × smaller than the spatial temporal graph attention network (STGAT) proposed in 2022 while retaining very competitive accuracy. The findings suggest that this model significantly improves the quality of feature extraction. As a result, DenseGCN presents a remarkable balance between accuracy and lightness.

Dynamic Fall Detection Using Graph-Based Spatial Temporal Convolution and Attention Network

The prevention of falls has become crucial in the modern healthcare domain and in society for improving ageing and supporting the daily activities of older people. Falling is mainly related to age and health problems such as muscle, cardiovascular, and locomotive syndrome weakness, etc. Among elderly people, the number of falls is increasing every year, and they can become life-threatening if detected too late. Most of the time, ageing people consume prescription medication after a fall and, in the Japanese community, the prevention of suicide attempts due to taking an overdose is urgent. Many researchers have been working to develop fall detection systems to observe and notify about falls in real-time using handcrafted features and machine learning approaches. Existing methods may face difficulties in achieving a satisfactory performance, such as limited robustness and generality, high computational complexity, light illuminations, data orientation, and camera view issues. We proposed a graph-based spatial-temporal convolutional and attention neural network (GSTCAN) with an attention model to overcome the current challenges and develop an advanced medical technology system. The spatial-temporal convolutional system has recently proven the power of its efficiency and effectiveness in various fields such as human activity recognition and text recognition tasks. In the procedure, we first calculated the motion along the consecutive frame, then constructed a graph and applied a graph-based spatial and temporal convolutional neural network to extract spatial and temporal contextual relationships among the joints. Then, an attention module selected channel-wise effective features. In the same procedure, we repeat it six times as a GSTCAN and then fed the spatial-temporal features to the network. Finally, we applied a softmax function as a classifier and achieved high accuracies of 99.93%, 99.74%, and 99.12% for ImViA, UR-Fall, and FDD datasets, respectively. The high-performance accuracy with three datasets proved the proposed system’s superiority, efficiency, and generality.

Despeckling of SAR Images Using Residual Twin CNN and Multi-Resolution Attention Mechanism

The despeckling of synthetic aperture radar images using two different convolutional neural network architectures is presented in this paper. The first method presents a novel Siamese convolutional neural network with a dilated convolutional network in each branch. Recently, attention mechanisms have been introduced to convolutional networks to better model and recognize features. Therefore, we propose a novel design for a convolutional neural network using an attention mechanism for an encoder–decoder-type network. The framework consists of a multiscale spatial attention network to improve the modeling of semantic information at different spatial levels and an additional attention mechanism to optimize feature propagation. Both proposed methods are different in design but they provide comparable despeckling results in subjective and objective measurements in terms of correlated speckle noise. The experimental results are evaluated on both synthetically generated speckled images and real SAR images. The methods proposed in this paper are able to despeckle SAR images and preserve SAR features.

Enhancing wheat Fusarium head blight detection using rotation Yolo wheat detection network and simple spatial attention network

The detection of Fusarium head blight (FHB), a destructive disease in wheat, can be performed through digit imaging. To improve detection accuracy and overcome challenges related to accurate labelling and detection efficacy, this study introduced two new networks: the Rotation Yolo Wheat Detection (RYWD) network and the Simple Spatial Attention (SSA) network. The RYWD network, utilizing the Yolo structure, served as a novel rotation detector capable of detecting wheat head images with detection boxes of arbitrary orientations. Angle prediction performance was enhanced by employing gray coding labels for angle encoding. Additionally, the SSA network, an unsupervised segmentation network, incorporated a spatial attention module and a spatial continuity loss to extract wheat features based on their spatial distribution. FHB detection was accomplished through HSV threshold segmentation and K-Means segmentation. The proposed method achieved an average accuracy of 94.66% in predicting the levels of FHB across two different years and locations. Comparatively, the proposed method outperformed previous research, exhibiting significant increases in both accuracy (11.8% increase) and precision (10.7% increase). These findings highlight the considerable improvement attainable through the integration of a rotation detector in crop disease detection, demonstrating its enhanced efficiency.

Distinct neural mechanisms construct classical versus extraclassical inhibitory surrounds in an inhibitory nucleus in the midbrain attention network

Inhibitory neurons in the midbrain spatial attention network, called isthmi pars magnocellularis (Imc), control stimulus selection by the sensorimotor and attentional hub, the optic tectum (OT). Here, we investigate in the barn owl how classical as well as extraclassical (global) inhibitory surrounds of Imc receptive fields (RFs), fundamental units of Imc computational function, are constructed. We find that focal, reversible blockade of GABAergic input onto Imc neurons disconnects their extraclassical inhibitory surrounds, but leaves intact their classical inhibitory surrounds. Subsequently, with paired recordings and iontophoresis, first at spatially aligned site-pairs in Imc and OT, and then, at mutually distant site-pairs within Imc, we demonstrate that classical inhibitory surrounds of Imc RFs are inherited from OT, but their extraclassical inhibitory surrounds are constructed within Imc. These results reveal key design principles of the midbrain spatial attention circuit and highlight the critical importance of competitive interactions within Imc for its operation.

Cascaded Temporal and Spatial Attention Network for solar adaptive optics image restoration

Context. Atmospheric turbulence severely degrades the quality of images observed through a ground-based telescope. An adaptive optics (AO) system only partially improves the image quality by correcting certain level wavefronts, making post-facto image processing necessary. Several deep learning-based methods have recently been applied in solar AO image post-processing. However, further research is still needed to get better images while enhancing model robustness and using inter-frame and intra-frame information. Aims. We propose an end-to-end network that can better handle solar adaptive image anisoplanatism by leveraging attention mechanisms, pixel-wise filters, and cascaded architecture. Methods. We developed a cascaded attention-based deep neural network named Cascaded Temporal and Spatial Attention Network (CTSAN) for solar AO image restoration. CTSAN consists of four modules: optical flow estimation PWC-Net for inter-frame explicit alignment, temporal and spatial attention for dynamic feature fusion, temporal sharpness prior for sharp feature extraction, and encoder-decoder architecture for feature reconstruction. We also used a hard example mining strategy to create a loss function in order to focus on the regions that are difficult to restore, and a cascaded architecture to further improve model stability. Results. CTSAN and the other two state-of-the-art (SOTA) supervised learning methods for solar AO image restoration are trained on real 705 nm photospheric and 656 nm chromospheric AO images supervised by corresponding Speckle images. Then all the methods are quantitatively and qualitatively evaluated on five real testing sets. Compared to the other two SOTA methods, CTSAN can restore clearer solar images, and shows better stability and generalization performance when restoring the lowest contrast AO image.

Densely Convolutional Spatial Attention Network for nuclei segmentation of histological images for computational pathology.

Automatic nuclear segmentation in digital microscopic tissue images can aid pathologists to extract high-quality features for nuclear morphometrics and other analyses. However, image segmentation is a challenging task in medical image processing and analysis. This study aimed to develop a deep learning-based method for nuclei segmentation of histological images for computational pathology. The original U-Net model sometime has a caveat in exploring significant features. Herein, we present the Densely Convolutional Spatial Attention Network (DCSA-Net) model based on U-Net to perform the segmentation task. Furthermore, the developed model was tested on external multi-tissue dataset - MoNuSeg. To develop deep learning algorithms for well-segmenting nuclei, a large quantity of data are mandatory, which is expensive and less feasible. We collected hematoxylin and eosin-stained image data sets from two hospitals to train the model with a variety of nuclear appearances. Because of the limited number of annotated pathology images, we introduced a small publicly accessible data set of prostate cancer (PCa) with more than 16,000 labeled nuclei. Nevertheless, to construct our proposed model, we developed the DCSA module, an attention mechanism for capturing useful information from raw images. We also used several other artificial intelligence-based segmentation methods and tools to compare their results to our proposed technique. To prioritize the performance of nuclei segmentation, we evaluated the model's outputs based on the Accuracy, Dice coefficient (DC), and Jaccard coefficient (JC) scores. The proposed technique outperformed the other methods and achieved superior nuclei segmentation with accuracy, DC, and JC of 96.4% (95% confidence interval [CI]: 96.2 - 96.6), 81.8 (95% CI: 80.8 - 83.0), and 69.3 (95% CI: 68.2 - 70.0), respectively, on the internal test data set. Our proposed method demonstrates superior performance in segmenting cell nuclei of histological images from internal and external datasets, and outperforms many standard segmentation algorithms used for comparative analysis.

A dual channel and spatial attention network for automatic spine segmentation of MRI images

AbstractAccurate image segmentation plays an essential role in diagnosing and treating various spinal diseases. However, traditional segmentation methods often consume a lot of time and energy. This research proposes an innovative deep‐learning‐based automatic segmentation method for spine magnetic resonance imaging (MRI) images. The proposed method DAUNet++ is supported by UNet++, which adds residual structure and attention mechanism. Specifically, a residual block is utilized for down‐sampling to construct the RVNet, as a new skeleton structure. Furthermore, two novel types of dual channel and spatial attention modules are proposed to emphasize rich feature regions, enhance useful information, and improve the network performance by recalibrating the characteristic. The published spinesagt2wdataset3 spinal MRI image dataset is adopted in the experiment. The dice similarity coefficient score on the test set is 0.9064. Higher segmentation accuracy and efficiency are achieved, indicating the effectiveness of the proposed method.

Resting State Dynamic Reconfiguration of Spatial Attention Cortical Networks and Visuospatial Functioning in Non-Verbal Learning Disability (NVLD): A HD-EEG Investigation.

Nonverbal learning disability (NVLD) is a neurodevelopmental disorder characterized by deficits in visuospatial processing but spared verbal competencies. Neurocognitive markers may provide confirmatory evidence for characterizing NVLD as a separate neurodevelopmental disorder. Visuospatial performance and high-density electroencephalography (EEG) were measured in 16 NLVD and in 16 typically developing (TD) children. Cortical source modeling was applied to assess resting-state functional connectivity (rs-FC) in spatial attention networks (dorsal (DAN) and ventral attention networks (VAN)) implicated in visuospatial abilities. A machine-learning approach was applied to investigate whether group membership could be predicted from rs-FC maps and if these connectivity patterns were predictive of visuospatial performance. Graph theoretical measures were applied to nodes inside each network. EEG rs-FC maps in the gamma and beta band differentiated children with and without NVLD, with increased but more diffuse and less efficient functional connections bilaterally in the NVLD group. While rs-FC of the left DAN in the gamma range predicted visuospatial scores for TD children, in the NVLD group rs-FC of the right DAN in the delta range predicted impaired visuospatial performance, confirming that NVLD is a disorder with a predominant dysfunction in right hemisphere connectivity patterns.

Emotion estimation model for cognitive state analysis of learners in online education using deep learning

AbstractFor Facial Expression Recognition, occlusion and position change that may drastically alter facial expressions are two important challenges (FER). Due to advances in automated FER over the last several decades, it has received less attention in the real world, where occlusion‐ and pose‐invariant aspects of FER are critical. Online education learners' cognitive states may be assessed using this paper's focus on real‐world stance and occlusion robust FER. The human visual system's attention mechanism inspired us to develop a new kind of spatial attention network (SAN‐CNN). Saliency characteristics and spatial importance between adjacent pixels are emphasized in the SAN‐CNN model. Preprocessing an input picture using a median contour filter is used here initially, followed by mask‐based ROI for segmentation. Using CNNs for facial recognition, landmark localization, and head position estimation based on spatial attention networks, it is possible to do emotional categorization. Using the Kaggle public video‐based facial expression datasets, we were able to demonstrate that our proposed approach is more accurate and faster than the usual techniques. In addition, we evaluated the suggested method's performance metrics with those of the already used approaches. FER with occlusion and variant posture performs better than standard approaches using our suggested method, as shown by the results of the experiments.

T-A-MFFNet: Multi-feature fusion network for EEG analysis and driving fatigue detection based on time domain network and attention network

Driving fatigue detection based on EEG signals is a research hotspot in applying brain-computer interfaces. EEG signal is complex, unstable, and nonlinear. Most existing methods rarely analyze the data characteristics from multiple dimensions, so it takes work to analyze the data comprehensively. To analyze EEG signals more comprehensively, this paper evaluates a feature extraction strategy of EEG data based on differential entropy (DE). This method combines the characteristics of different frequency bands, extracts the frequency domain characteristics of EEG, and retains the spatial information between channels. This paper proposes a multi-feature fusion network (T-A-MFFNet) based on the time domain and attention network. The model is composed of a time domain network (TNet), channel attention network (CANet), spatial attention network (SANet), and multi-feature fusion network(MFFNet) based on a squeeze network. T-A-MFFNet aims to learn more valuable features from the input data to achieve good classification results. Specifically, the TNet network extracts high-level time series information from EEG data. CANet and SANet are used to fuse channel and spatial features. They use MFFNet to merge multi-dimensional features and realize classification. The validity of the model is verified on the SEED-VIG dataset. The experimental results show that the accuracy of the proposed method reaches 85.65 %, which is superior to the current popular model. The proposed method can learn more valuable information from EEG signals to improve the ability to identify fatigue status and promote the development of the research field of driving fatigue detection based on EEG signals.

MSA-Net: Multiscale spatial attention network for medical image segmentation

BackgroundEdge accuracy and positional accuracy are the two goals pursued by medical image segmentation. In clinical medicine diagnosis and research, these two goals enable medical image segmentation techniques to help in the effective determination of lesions and lesion analysis. At present, U-Net has become the most important network in the field of image segmentation, and the technologies used in various achievements are derived from its architecture, which also proves from practice that the network structure proposed by U-Net is effective. ObjectiveWe have found in a large number of experiments that classical networks indeed show good performance in the field of medical segmentation, but there are still some deficiencies in edge determination and network robustness, especially in the face of blurred edges, the processing results often fail to achieve the expected results. In order to be able to locate segmentation targets and achieve effective determination of blurred edges, a Multiscale Spatial Attention Network (MSA-Net) is proposed as in Fig. 1. MethodIn MSA-Net, the Multiscale Pyramid Attention Block (MPAB) is created to enhance the capture of high-level semantic information. In addition, the network uses ASPP, which not only expands the network’s field of view, but also captures richer feature information. In the decoding phase, the Feature Fusion Block (FFB) is created to enable better focus on different dimensional information features and to enhance the feature fusion process. ResultTo demonstrate the effectiveness of the network, we validate the performance of MSA-Net on four datasets (ISIC2016, DSB2018, JSRT, GlaS) in three different categories. Compared with mainstream networks, MSA-Net shows better results in detail features, target localization, and edge processing. Finally, we also demonstrate the effectiveness of the MSA-Net architecture through ablation experiments.

Joint spatial and scale attention network for multi-view facial expression recognition

Multi-view facial expression recognition (FER) is a challenging task because the appearance of an expression varies greatly due to poses. To alleviate the influences of poses, recently developed methods perform pose normalization, learn pose-invariant features, or learn pose-specific FER classifiers. However, these methods usually rely on a prerequisite pose estimator or expressive region detector that is independent of the subsequent expression analysis. Different from existing methods, we propose a joint spatial and scale attention network (SSA-Net) to localize proper regions for simultaneous head pose estimation (HPE) and FER. Specifically, SSA-Net discovers the regions most relevant to the facial expression at hierarchical scales by a spatial attention mechanism, and the most informative scales are selected in a scale attention learning manner to learn the joint pose-invariant and expression-discriminative representations. Then, we employ a dynamically constrained multi-task learning mechanism with a delicately designed constrain regulation to properly and adaptively train the network to optimize the representations, thus achieving accurate multi-view FER. The effectiveness of the proposed SSA-Net is validated on three multi-view datasets (BU-3DFE, Multi-PIE, and KDEF) and three in-the-wild FER datasets (AffectNet, SFEW, and FER2013). Extensive experiments demonstrate that the proposed framework outperforms existing state-of-the-art methods under both within-dataset and cross-dataset settings, with relative accuracy gains of 2.36%, 1.33%, 3.11%, 2.84%, 15.7%, and 7.57%, respectively.

TASTA: Text‐Assisted Spatial and Temporal Attention Network for Video Question Answering

Video question answering (VideoQA) is a typical task that integrates language and vision. The key for VideoQA is to extract relevant and effective visual information for answering a specific question. Information selection is believed to be necessary for this task due to the large amount of irrelevant information in the video, and explicitly learning an attention model can be a reasonable and effective solution for the selection. Herein, a novel VideoQA model called Text‐Assisted Spatial and Temporal Attention Network (TASTA) is proposed, which shows the great potential of explicitly modeling attention. TASTA is made to be simple, small, clean, and efficient for clear performance justification and possible easy extension. Its success is mainly from two new strategies of better using the textual information. Experimental results on a large and most representative dataset, TGIF‐QA, show the significant superiority of TASTA w.r.t. the state‐of‐the‐art and demonstrate the effectiveness of its key components via ablation studies.