Multi-scale Attention Network Research Articles

Adaptive Aggregated Attention Network for Pulmonary Nodule Classification

Lung cancer has one of the highest cancer mortality rates in the world and threatens people’s health. Timely and accurate diagnosis can greatly reduce the number of deaths. Therefore, an accurate diagnosis system is extremely important. The existing methods have achieved significant performances on lung cancer diagnosis, but they are insufficient in fine-grained representations. In this paper, we propose a novel attentive method to differentiate malignant and benign pulmonary nodules. Firstly, the residual attention network (RAN) and squeeze-and-excitation network (SEN) were utilized to extract spatial and contextual features. Secondly, a novel multi-scale attention network (MSAN) was proposed to capture multi-scale attention features automatically, and the MSAN integrated the advantages of the spatial attention mechanism and contextual attention mechanism, which are very important for capturing the salient features of nodules. Finally, the gradient boosting machine (GBM) algorithm was used to differentiate malignant and benign nodules. We conducted a series of experiments on the Lung Image Database Consortium image collection (LIDC-IDRI) database, achieving an accuracy of 91.9%, a sensitivity of 91.3%, a false positive rate of 8.0%, and an F1-score of 91.0%. The experimental results demonstrate that our proposed method outperforms the state-of-the-art methods with respect to accuracy, false positive rate, and F1-Score.

An Efficient Video Coding System With an Adaptive Overfitted Multi-Scale Attention Network

We herein propose an efficient video coding system (EVCS) consists of a conventional codec and an adaptive overfitted multi-scale attention network (MSAN) to improve coding efficiency. At the encoder side, the MSAN adjusts the network size adaptively and is trained in an overfitting way for a group of frames. Using only the current encoding video stream as a training set, the MSAN can easily obtain powerful restoration capability. After training, the learned parameters of the MSAN are transmitted to the decoder as part of the encoded bitstream. At the decoder side, the MSAN loaded with the transmitted parameters can restore the reconstructed frames very meticulously. Compared with the high efficiency video coding (HEVC) standard, the EVCS can achieve 12.141% Bjontegaard-Delta bitrate reduction, which outperforms existing deep learning based compressed video restoration works with less computation complexity. Moreover, the MSAN is an additional part of the conventional codec without any structure change, thereby rendering it compatible with the existing coding systems.

Multi-Scale Attention Network for Diabetic Retinopathy Classification

Diabetic Retinopathy (DR) is a highly prevalent complication of diabetes mellitus, which causes lesions on the retina that affect vision which may lead to blindness if it is not detected and diagnosed early. Convolutional neural networks (CNN) are becoming the state-of-the-art approach for automatic detection of DR by using fundus images. The high-level features extracted by CNN are mostly utilised for the detection and classification of lesions on the retina. This high-level representation is capable of classifying different DR classes; however, more effective features for detecting the damages are needed. This paper proposes the multi-scale attention network (MSA-Net) for DR classification. The proposed approach applies the encoder network to embed the retina image in a high-level representational space, where the combination of mid and high-level features is used to enrich the representation. Then a multi-scale feature pyramid is included to describe the retinal structure in a different locality. Furthermore, to enhance the discriminative power of the feature representation a multi-scale attention mechanism is used on top of the high-level representation. The model is trained in a standard way using the cross-entropy loss to classify the DR severity level. In parallel as an auxiliary task, the model is trained using the weakly annotated data to detect healthy and non-healthy retina images. This surrogate task helps the model to enrich its discriminative power for distinguishing the non-healthy retina images. The proposed method when implemented has achieved outstanding results on two public datasets: EyePACS and APTOS.

Multi-Scale Convolutional Neural Networks for Classification of Digital Mammograms With Breast Calcifications

Breast cancer is the most commonly diagnosed cancer in women. Mammography is a widely used tool for breast cancer screening. Breast calcifications appear on mammograms as tiny white spots or grainy dots with a variety of clustering structures which may be time-consuming and difficult to identify by the human eye. Recent advances in multi-scale architectures have demonstrated strong feature representation for both large-scale contextual information and small-scale features. We hypothesized that the new architectures could be employed for classification of mammograms with breast calcifications and possibly for localization of breast calcification regions. In this study, we investigated different multi-scale architectures on 1,617 mammograms that contained only breast calcifications locally curated from our institution. Our best multi-scale attention network with hierarchical block-wise and layer-wise feature representation capability achieved an accuracy of 84.34% and an area under the receiver operating characteristic curve (AUROC) of 90.36%. Similar performance was observed across breast density categories. Our network simultaneously processes both craniocaudal (CC) and mediolateral oblique (MLO) views of the breast providing class activation maps that were able to precisely localize breast calcification regions in a weakly supervised manner, indicating the benefits of strong feature representation through the multi-scale architectures.

基于多尺度注意力网络的行人属性识别算法

基于多尺度注意力网络的行人属性识别算法

MANet: Multi-Scale Attention Network for Correspondence Learning

Establishing reliable correspondences from a putative correspondence set is a challenging task. Most of state-of-the-art methods utilize the local context and global context to address the task. However, the local and global context often contains large number of outliers, which have a negative impact on capturing scene geometry. In this paper, we propose a Multi-scale Attention Network (called MANet), which introduces the attention mechanism for feature matching, to improve the ability of capturing scene geometry. Specifically, we first fuse the features of low and high levels by an multi-scale strategy network to enhance the representative ability of features. Then, we propose an attentive PointCN block and an attentive pooling layer, to discriminatively capture global context and local context information, respectively. We demonstrate through extensive experiments on both indoor and outdoor datasets that MANet provides a significant improvement in the performance of the two-view geometry and correspondences accuracy compared to the state-of-the-art methods.

Uncertainty Guided Multi-Scale Attention Network for Raindrop Removal From a Single Image.

Raindrops adhered to a glass window or camera lens appear in various blurring degrees and resolutions due to the difference in the degrees of raindrops aggregation. The removal of raindrops from a rainy image remains a challenging task because of the density and diversity of raindrops. The abundant location and blur level information are strong prior guide to the task of raindrop removal. However, existing methods use a binary mask to locate and estimate the raindrop with the value 1 (adhesion of raindrops) and 0 (no adhesion), which ignores the diversity of raindrops. Meanwhile, it is noticed that different scale versions of a rainy image have similar raindrop patterns, which makes it possible to employ such complementary information to represent raindrops. In this work, we first propose a soft mask with the value in [-1,1] indicating the blurring level of the raindrops on the background, and explore the positive effect of the blur degree attribute of raindrops on the task of raindrop removal. Secondly, we explore the multi-scale fusion representation for raindrops based on the deep features of the input multi-scale images. The framework is termed uncertainty guided multi-scale attention network (UMAN). Specifically, we construct a multi-scale pyramid structure and introduce an iterative mechanism to extract blur-level information about raindrops to guide the removal of raindrops at different scales. We further introduce the attention mechanism to fuse the input image with the blur-level information, which will highlight raindrop information and reduce the effects of redundant noise. Our proposed method is extensively evaluated on several benchmark datasets and obtains convincing results.

Densely Connected Multiscale Attention Network for Hyperspectral Image Classification

Hyperspectral images (HSIs) are characterized by high spatial resolution and are rich in spectral information. In the process of HSI classification, the extraction of spectral-spatial features directly influences the classification results. In recent years, the hyperspectral classification method based on convolutional neural networks has demonstrated excellent performance. However, as the network structure deepens, degradation occurs, and the features learned from the fixed-scale convolutional kernels are usually specific, which is not conducive to feature learning and thus impairs the classification accuracy. To solve the problem of difficult extraction of features and underutilization of information from HSI data, a densely connected multiscale attention network based on 3-D convolution is proposed for HSI classification. First, to reduce the spectral redundancy of the HSIs, the principal component analysis algorithm is performed on the raw HSI data; then, several multiscale blocks comprised of parallel factorized spatial-spectral convolution modules of different sizes are adopted to extract the enriched spectral-spatial features from HSIs; furthermore, dense connections are introduced to further fuse features obtained from blocks of different depths, thereby enhancing feature reuse and propagation and helping to alleviate the problem of vanishing gradients. Besides, the channel-spectral-spatial attention block is put forward to spontaneously reweight the fused features to emphasize the features that are more relevant to the classification results while weakening the less relevant ones. The experimental results show that the proposed method is effective in extracting discriminative features of the target and outperforms the other state-of-the-art methods.

A Multiscale Attention Network for Remote Sensing Scene Images Classification

The remote sensing scene images classification has been of great value to civil and military fields. Deep learning models, especially the convolutional neural network (CNN), have achieved great success in this task, however, they may suffer from two challenges: firstly, the sizes of the category objects are usually different, but the conventional CNN extracts the features with fixed convolution extractor which could cause the failure in learning the multi-scale features; secondly, some image regions may not be useful during the feature learning process, therefore, how to guide the network to select and focus on the most relevant regions is crucially vital for remote sensing scene image classification. To address these two challenges, we propose a multi-scale attention network (MSA-Network), which integrates a multi-scale (MS) module and a channel and position attention (CPA) module to boost the performance of the remote sensing scene classification. The proposed MS module learns multi-scale features by adopting various sizes of sliding windows from different depths layers and receptive fields. The CPA module is composed of two parts: the channel attention (CA) module and position attention (PA) one. The CA module learns the global attention features from channel-level, and the PA module extracts the local attention features from pixel-level. Thus, fusing both of those two attention features, the network is apt to focus on the more critical and salient regions automatically. Extensive experiments on UC Merced, AID, NWPU-RESISC45 datasets demonstrate that the proposed MSA-Network outperforms several state-of-the-art methods.

Attention to fine-grained information: hierarchical multi-scale network for retinal vessel segmentation

Medical segmentation is a task that pays attention to details. The rapid development of deep learning in image processing technology makes it possible to segment objects accurately on small datasets. In this paper, we propose a hierarchical multi-scale attention network that focuses on the fine-grained parts of the target. Our attention network consists of a hierarchical encoder module with dense connections, a multi-scale module attention to fine-grained parts, and a decoder module. We also combine the weighted cross-entropy loss function based on details and the Dice coefficient loss to increase the sensitivity of fine grains. To verify our module’s performance, we carried out a series of comparative experiments on the multi-scale attention module on the DRIVE dataset. We determine the best structure through experiments and compare it with several classical deep learning models. Our experiments show that extracting semantic information of images at an appropriate resolution can also improve the accuracy of detail segmentation. To show the generalization ability of the work, we conducted experiments on different DRIVE, STARE, and CHASE_DB 1 datasets, and our method achieved 0.8802/0.8464/0.8216 in sensitivity performance metric, 0.9756/0.9869/0.9784 in specificity, and 0.9675/0.9657/0.9637 in accuracy.

Multi-scale attention network for image inpainting

Recently, deep learning based inpainting methods have shown promising performance, in which some multi-scale networks are introduced to characterize image content in both details and structures. However, few of these networks explore local spatial components under different receptive fields and internal connection between multi-scale feature maps. In this paper, we propose a novel multi-scale attention network (MSA-Net) to fill the irregular missing regions, in which a multi-scale attention group (MSAG) with several multi-scale attention units (MSAUs) is introduced for fully analysing the features from shallow details to high-level semantics. In each MSAU, an attention based spatial pyramid structure is designed to capture the deep features from different receptive fields. In this network, attention mechanisms are explored for boosting the representation power of MSAU, where spatial attention is combined with each scale to highlight the most probably attentive spatial components and the channel attention is used as a globally semantic detector to build the connection between the multiple scales. Furthermore, for better inpainting results, a max pooling based mask update method is utilized to predict the missing parts from the border regions to the inside. Finally, experiments on Places2 dataset and CelebA dataset demonstrate that the proposed method can achieve better results than the previous inpainting methods.

Multi-Scale Dense Attention Network for Stereo Matching

To improve the accuracy of stereo matching, the multi-scale dense attention network (MDA-Net) is proposed. The network introduces two novel modules in the feature extraction stage to achieve better exploit of context information: dual-path upsampling (DU) block and attention-guided context-aware pyramid feature extraction (ACPFE) block. The DU block is introduced to fuse different scale feature maps. It introduces sub-pixel convolution to compensate for the loss of information caused by the traditional interpolation upsampling method. The ACPFE block is proposed to extract multi-scale context information. Pyramid atrous convolution is adopted to exploit multi-scale features and the channel-attention is used to fuse the multi-scale features. The proposed network has been evaluated on several benchmark datasets. The three-pixel-error evaluated over all ground truth pixels is 2.10% on KITTI 2015 dataset. The experiment results prove that MDA-Net achieves state-of-the-art accuracy on KITTI 2012 and 2015 datasets.

Scene-Adaptive Remote Sensing Image Super-Resolution Using a Multiscale Attention Network

Remote sensing image super-resolution has always been a major research focus, and many deep-learning-based algorithms have been proposed in recent years. However, since the structure of remote sensing images tends to be much more complex than that of natural images, several difficulties still remain for remote sensing images super-resolution. First, it is difficult to depict the nonlinear mapping between high-resolution (HR) and low-resolution (LR) images of different scenes with the same model. Second, the wide range of scales within the ground objects in remote sensing images makes it difficult for single-scale convolution to effectively extract features of various scales. To address the above-mentioned issues, we propose a multiscale attention network (MSAN) to extract the multilevel features of remote sensing images. The basic component of MSAN is the multiscale activation feature fusion block (MAFB). In addition, a scene-adaptive super-resolution strategy for remote sensing images is employed to more accurately describe the structural characteristics of different scenes. The experiments undertaken on several data sets confirm that the proposed algorithm outperforms the other state-of-the-art algorithms, in both evaluation indices and visual results.

Weak supervision for generating pixel–level annotations in scene text segmentation

Providing pixel–level supervisions for scene text segmentation is inherently difficult and costly, so that only few small datasets are available for this task. To face the scarcity of training data, previous approaches based on Convolutional Neural Networks (CNNs) rely on the use of a synthetic dataset for pre–training. However, synthetic data cannot reproduce the complexity and variability of natural images. In this work, we propose to use a weakly supervised learning approach to reduce the domain–shift between synthetic and real data. Leveraging the bounding–box supervision of the COCO–Text and the MLT datasets, we generate weak pixel–level supervisions of real images. In particular, the COCO–Text–Segmentation (COCO_TS) and the MLT–Segmentation (MLT_S) datasets are created and released. These two datasets are used to train a CNN, the Segmentation Multiscale Attention Network (SMANet), which is specifically designed to face some peculiarities of the scene text segmentation task. The SMANet is trained end–to–end on the proposed datasets, and the experiments show that COCO_TS and MLT_S are a valid alternative to synthetic images, allowing to use only a fraction of the training samples, with a significant improvement in performance.

A Novel Attention Fully Convolutional Network Method for Synthetic Aperture Radar Image Segmentation

As an important step of synthetic aperture radar image interpretation, synthetic aperture radar image segmentation aims at segmenting an image into different regions in terms of homogeneity. Because of the deficiency of the labeled samples and the existence of speckling noise, synthetic aperture radar image segmentation is a challenging task. We present a new method for synthetic aperture radar image segmentation in this article. Due to the large size of the original synthetic aperture radar image, we first divide the input image into small slices. Then the image slices are input to the attention-based fully convolutional network for obtaining the segmentation results. Finally, the fully connected conditional random field is adopted for improving the segmentation performance of the network. The innovations of our method are as follows: 1) The attention-based fully convolutional network is embedded with the multiscale attention network which is capable of enhancing the extraction of the image features through three strategies, namely, multiscale feature extraction, channel attention extraction, and spatial attention extraction. 2) We design a new loss function for the attention fully convolutional network by combining Lovasz-Softmax and cross-entropy losses. The new loss allows us to simultaneously optimize the intersection over union and the pixel classification accuracy of the segmentation results. The experiments are performed on two airborne synthetic aperture radar image databases. It has been proved that our method is superior to other state-of- the-art image segmentation approaches.

基于多尺度残差注意力网络的壁画图像超分辨率重建算法

基于多尺度残差注意力网络的壁画图像超分辨率重建算法

MA-Net: A Multi-Scale Attention Network for Liver and Tumor Segmentation

Automatic assessing the location and extent of liver and liver tumor is critical for radiologists, diagnosis and the clinical process. In recent years, a large number of variants of U-Net based on Multi-scale feature fusion are proposed to improve the segmentation performance for medical image segmentation. Unlike the previous works which extract the context information of medical image via applying the multi-scale feature fusion, we propose a novel network named Multi-scale Attention Net (MA-Net) by introducing self-attention mechanism into our method to adaptively integrate local features with their global dependencies. The MA-Net can capture rich contextual dependencies based on the attention mechanism. We design two blocks: Position-wise Attention Block (PAB) and Multi-scale Fusion Attention Block (MFAB). The PAB is used to model the feature interdependencies in spatial dimensions, which capture the spatial dependencies between pixels in a global view. In addition, the MFAB is to capture the channel dependencies between any feature map by multi-scale semantic feature fusion. We evaluate our method on the dataset of MICCAI 2017 LiTS Challenge. The proposed method achieves better performance than other state-of-the-art methods. The Dice values of liver and tumors segmentation are 0.960 ± 0.03 and 0.749 ± 0.08 respectively.

Pulmonary Textures Classification via a Multi-Scale Attention Network.

Precise classification of pulmonary textures is crucial to develop a computer aided diagnosis (CAD) system of diffuse lung diseases (DLDs). Although deep learning techniques have been applied to this task, the classification performance is not satisfied for clinical requirements, since commonly-used deep networks built by stacking convolutional blocks are not able to learn discriminative feature representation to distinguish complex pulmonary textures. For addressing this problem, we design a multi-scale attention network (MSAN) architecture comprised by several stacked residual attention modules followed by a multi-scale fusion module. Our deep network can not only exploit powerful information on different scales but also automatically select optimal features for more discriminative feature representation. Besides, we develop visualization techniques to make the proposed deep model transparent for humans. The proposed method is evaluated by using a large dataset. Experimental results show that our method has achieved the average classification accuracy of 94.78% and the average f-value of 0.9475in the classification of 7 categories of pulmonary textures. Besides, visualization results intuitively explain the working behavior of the deep network. The proposed method has achieved the state-of-the-art performance to classify pulmonary textures on high resolution CT images.

Hierarchical Multi-scale Attention Networks for action recognition

Recurrent Neural Networks (RNNs) have been widely used in natural language processing and computer vision. Amongst them, the Hierarchical Multi-scale RNN (HM-RNN), a recently proposed multi-scale hierarchical RNN, can automatically learn the hierarchical temporal structure from data. In this paper, we extend the work to solve the computer vision task of action recognition. However, in sequence-to-sequence models like RNN, it is normally very hard to discover the relationships between inputs and outputs given static inputs. As a solution, the attention mechanism can be applied to extract the relevant information from the inputs thus facilitating the modeling of the input–output relationships. Based on these considerations, we propose a novel attention network, namely Hierarchical Multi-scale Attention Network (HM-AN), by incorporating the attention mechanism into the HM-RNN and applying it to action recognition. A newly proposed gradient estimation method for stochastic neurons, namely Gumbel-softmax, is exploited to implement the temporal boundary detectors and the stochastic hard attention mechanism. To reduce the negative effect of the temperature sensitivity of the Gumbel-softmax, an adaptive temperature training method is applied to improve the system performance. The experimental results demonstrate the improved effect of HM-AN over LSTM with attention on the vision task. Through visualization of what has been learnt by the network, it can be observed that both the attention regions of the images and the hierarchical temporal structure can be captured by a HM-AN.