Multi-scale Attention Mechanism Research Articles

Object Detection in Large-Scale Remote-Sensing Images Based on Time-Frequency Analysis and Feature Optimization

Recently, optical remote-sensing images have been steadily growing in size, as they contain massive data and complex backgrounds. This trend presents several problems for object detection, for example, increased computation time and memory consumption and more false positives due to the complex backgrounds of large-scale images. Inspired by deep neural networks combined with time-frequency analysis, we propose a time-frequency analysis-based object detection method for large-scale remote-sensing images with complex backgrounds. We utilize wavelet decomposition to carry out a time-frequency transform and then integrate it with deep learning in feature optimization. To effectively capture the time-frequency features, we propose a feature optimization method based on deep reinforcement learning to select the dominant time-frequency channels. Furthermore, we design a discrete wavelet multiscale attention mechanism (DW-MAM), enabling the detector to concentrate on the object area rather than the background. Extensive experiments show that the proposed method of learning from time-frequency channels not only solves the challenges of large-scale and complex backgrounds, but also improves the performance compared to the original state-of-the-art object detection methods. In addition, the proposed method can be used with almost all object detection neural networks, regardless of whether they are anchor-based or anchor-free detectors, horizontal or rotation detectors.

A novel 1-D densely connected feature selection convolutional neural network for heart sounds classification.

BackgroundHeart sound auscultation, due to it being a non-invasive, convenient, and relatively low-cost technique, remains the dominant method for detection of cardiovascular disease.MethodsIn this paper, we present a method for identifying abnormal heart sounds based on a novel Dense Feature Selection Convolution Network framework (Dense-FSNet). The Dense-FSNet is comprised of multiple, circular dense connectivity modules, called Clique Blocks. These Clique Blocks can allow low-level and high-level features to stimulate each other for cyclic updates, which subsequently enhances the information flow among layers. Inspired by the channel-wise attention mechanism from Squeeze-and-Excitation Networks (SENet), we adopted squeeze-and-excitation block to avoid the progressive growth of parameters. The effect of the model was assessed using the accuracy, specificity, sensitivity, and area under the curve (AUC) values. To improve model performance, in addition to the structures mentioned above, we incorporated a multi-scale attention mechanism into our framework.ResultsUsing this attention mechanism, our model was able to achieve adaptively spatial feature fusion by adjusting a hyper-feature that contains higher level visual information and lower-level features including edge details and context information. It is worth noting that data balance technology was also used in the process of building the model, and notable results have been achieved.ConclusionsExperience using the PhysioNet/CinC 2016 dataset shows that our proposed Dense-FSNet models achieve state of the art levels in the classification and detection of abnormal heart sounds.

MAResNet: predicting transcription factor binding sites by combining multi-scale bottom-up and top-down attention and residual network.

Accurate identification of transcription factor binding sites is of great significance in understanding gene expression, biological development and drug design. Although a variety of methods based on deep-learning models and large-scale data have been developed to predict transcription factor binding sites in DNA sequences, there is room for further improvement in prediction performance. In addition, effective interpretation of deep-learning models is greatly desirable. Here we present MAResNet, a new deep-learning method, for predicting transcription factor binding sites on 690 ChIP-seq datasets. More specifically, MAResNet combines the bottom-up and top-down attention mechanisms and a state-of-the-art feed-forward network (ResNet), which is constructed by stacking attention modules that generate attention-aware features. In particular, the multi-scale attention mechanism is utilized at the first stage to extract rich and representative sequence features. We further discuss the attention-aware features learned from different attention modules in accordance with the changes as the layers go deeper. The features learned by MAResNet are also visualized through the TMAP tool to illustrate that the method can extract the unique characteristics of transcription factor binding sites. The performance of MAResNet is extensively tested on 690 test subsets with an average AUC of 0.927, which is higher than that of the current state-of-the-art methods. Overall, this study provides a new and useful framework for the prediction of transcription factor binding sites by combining the funnel attention modules with the residual network.

U-Net combined with multi-scale attention mechanism for liver segmentation in CT images

BackgroundThe liver is an important organ that undertakes the metabolic function of the human body. Liver cancer has become one of the cancers with the highest mortality. In clinic, it is an important work to extract the liver region accurately before the diagnosis and treatment of liver lesions. However, manual liver segmentation is a time-consuming and boring process. Not only that, but the segmentation results usually varies from person to person due to different work experience. In order to assist in clinical automatic liver segmentation, this paper proposes a U-shaped network with multi-scale attention mechanism for liver organ segmentation in CT images, which is called MSA-UNet. Our method makes a new design of U-Net encoder, decoder, skip connection, and context transition structure. These structures greatly enhance the feature extraction ability of encoder and the efficiency of decoder to recover spatial location information. We have designed many experiments on publicly available datasets to show the effectiveness of MSA-UNet. Compared with some other advanced segmentation methods, MSA-UNet finally achieved the best segmentation effect, reaching 98.00% dice similarity coefficient (DSC) and 96.08% intersection over union (IOU).

Dual Branch Attention Network for Person Re-Identification.

As a sub-direction of image retrieval, person re-identification (Re-ID) is usually used to solve the security problem of cross camera tracking and monitoring. A growing number of shopping centers have recently attempted to apply Re-ID technology. One of the development trends of related algorithms is using an attention mechanism to capture global and local features. We notice that these algorithms have apparent limitations. They only focus on the most salient features without considering certain detailed features. People’s clothes, bags and even shoes are of great help to distinguish pedestrians. We notice that global features usually cover these important local features. Therefore, we propose a dual branch network based on a multi-scale attention mechanism. This network can capture apparent global features and inconspicuous local features of pedestrian images. Specifically, we design a dual branch attention network (DBA-Net) for better performance. These two branches can optimize the extracted features of different depths at the same time. We also design an effective block (called channel, position and spatial-wise attention (CPSA)), which can capture key fine-grained information, such as bags and shoes. Furthermore, based on ID loss, we use complementary triplet loss and adaptive weighted rank list loss (WRLL) on each branch during the training process. DBA-Net can not only learn semantic context information of the channel, position, and spatial dimensions but can integrate detailed semantic information by learning the dependency relationships between features. Extensive experiments on three widely used open-source datasets proved that DBA-Net clearly yielded overall state-of-the-art performance. Particularly on the CUHK03 dataset, the mean average precision (mAP) of DBA-Net achieved 83.2%.

6mAPred-MSFF: A Deep Learning Model for Predicting DNA N6-Methyladenine Sites across Species Based on a Multi-Scale Feature Fusion Mechanism

DNA methylation is one of the most extensive epigenetic modifications. DNA N6-methyladenine (6mA) plays a key role in many biology regulation processes. An accurate and reliable genome-wide identification of 6mA sites is crucial for systematically understanding its biological functions. Some machine learning tools can identify 6mA sites, but their limited prediction accuracy and lack of robustness limit their usability in epigenetic studies, which implies the great need of developing new computational methods for this problem. In this paper, we developed a novel computational predictor, namely the 6mAPred-MSFF, which is a deep learning framework based on a multi-scale feature fusion mechanism to identify 6mA sites across different species. In the predictor, we integrate the inverted residual block and multi-scale attention mechanism to build lightweight and deep neural networks. As compared to existing predictors using traditional machine learning, our deep learning framework needs no prior knowledge of 6mA or manually crafted sequence features and sufficiently capture better characteristics of 6mA sites. By benchmarking comparison, our deep learning method outperforms the state-of-the-art methods on the 5-fold cross-validation test on the seven datasets of six species, demonstrating that the proposed 6mAPred-MSFF is more effective and generic. Specifically, our proposed 6mAPred-MSFF gives the sensitivity and specificity of the 5-fold cross-validation on the 6mA-rice-Lv dataset as 97.88% and 94.64%, respectively. Our model trained with the rice data predicts well the 6mA sites of other five species: Arabidopsis thaliana, Fragaria vesca, Rosa chinensis, Homo sapiens, and Drosophila melanogaster with a prediction accuracy 98.51%, 93.02%, and 91.53%, respectively. Moreover, via experimental comparison, we explored performance impact by training and testing our proposed model under different encoding schemes and feature descriptors.

CFFNet: Coordinated feature fusion network for crowd counting

Long skip connection or encoder-decoder networks for crowd counting have proven to be effective methods to generate high-resolution density maps. However, the simple and coarse feature fusion ignores the disharmony between features, that is, spatial misalignment and semantic inconsistency, which will weaken feature representation and degrade network performance. In this paper, we propose an end-to-end trainable architecture called Coordinated Feature Fusion Network (CFFNet) to tackle the aforementioned problems. The proposed model contains a powerful baseline network and embeds two primary modules: Spatial Alignment Module (SAM) and Semantic Consistency Module (SCM). Specifically, the SAM can learn the transformation offset of pixels to alleviate the spatial misalignment caused by the feature resolution difference; the SCM based on the multi-scale attention mechanism can capture pixel-wise weight to alleviate the semantic inconsistency due to the feature level gap. Extensive experiments on four benchmark crowd datasets (the ShanghaiTech, the UCF-QNRF, the JHU-CRWORD++ and the NWPU-Crowd), indicate that CFFNet can achieve state-of-the-art counting performance and high robustness.

Multiscale attention-based LSTM for ship motion prediction

Ship motion prediction is applied to the shipboard stabilized platform to keep the equipment on the platform stable all the time, which is of great practical significance to the safety and efficiency of shipboard equipment operation. Long Short-term Memory (LSTM) Network is a classic time series prediction method that has made remarkable achievements in this field. However, the dynamic frequency range of single LSTM in ship motion prediction is insufficient to meet the stabilized platform with higher precision requirements. To improve the performance of LSTM in ship motion prediction, this paper presents a novel method named as multiscale attention-based LSTM. At first, wavelet transform is employed to decompose ship motion signals into several frequency scales, which makes LSTM to capture the inherent law of ship motion from each frequency scale. And then the weights of different scales are obtained by attention mechanism, which promote the sensitivity of the whole system by paying more attention to significant information and suppress the interference of noise signals. Both of the steps form a multiscale attention mechanism, which promote the adaptability and improve the performance of the LSTM. In addition, to avoid being trapped in local optimization, the two-stage training mechanism is designed for model training based on the model structure. Ship motion data are used to evaluate the feasibility and effectiveness. The experiments show that the proposed method achieves better performance compared with other popular methods.

An Attention-Enhanced Multi-Scale and Dual Sign Language Recognition Network Based on a Graph Convolution Network.

Sign language is the most important way of communication for hearing-impaired people. Research on sign language recognition can help normal people understand sign language. We reviewed the classic methods of sign language recognition, and the recognition accuracy is not high enough because of redundant information, human finger occlusion, motion blurring, the diversified signing styles of different people, and so on. To overcome these shortcomings, we propose a multi-scale and dual sign language recognition Network (SLR-Net) based on a graph convolutional network (GCN). The original input data was RGB videos. We first extracted the skeleton data from them and then used the skeleton data for sign language recognition. SLR-Net is mainly composed of three sub-modules: multi-scale attention network (MSA), multi-scale spatiotemporal attention network (MSSTA) and attention enhanced temporal convolution network (ATCN). MSA allows the GCN to learn the dependencies between long-distance vertices; MSSTA can directly learn the spatiotemporal features; ATCN allows the GCN network to better learn the long temporal dependencies. The three different attention mechanisms, multi-scale attention mechanism, spatiotemporal attention mechanism, and temporal attention mechanism, are proposed to further improve the robustness and accuracy. Besides, a keyframe extraction algorithm is proposed, which can greatly improve efficiency by sacrificing a little accuracy. Experimental results showed that our method can reach 98.08% accuracy rate in the CSL-500 dataset with a 500-word vocabulary. Even on the challenging dataset DEVISIGN-L with a 2000-word vocabulary, it also reached a 64.57% accuracy rate, outperforming other state-of-the-art sign language recognition methods.

An improved one-stage pedestrian detection method based on multi-scale attention feature extraction

In recent years, the performance of the convolutional neural network-based pedestrian detection method has improved significantly. However, an imbalance remains between detection accuracy and speed. In this paper, we employ a one-stage object detection framework and propose a pedestrian detection method based on the multi-scale attention mechanism of a convolutional neural network to improve the imbalance between accuracy and speed. First, a multi-scale convolution module is designed to extract corresponding features at different scales. Second, using the attention module, association information between features is mined from space and channel perspectives to strengthen the original features. Then, the enhanced features are passed through a classification and regression module to perform object positioning and bounding box regression. Finally, to learn more pedestrian location information, we improve the loss function to realise better network training. The proposed method achieved considerable results on the challenging CityPersons and Caltech pedestrian detection datasets.

Towards accurate coronary artery calcium segmentation with multi‐scale attention mechanism

Coronary artery calcium is a strong and independent marker of atherosclerosis and cardiovascular disease. Typically, the accurate segmentation of computed tomography images of the chest is an important prerequisite and basis for coronary artery calcium identification and analysis. However, this is very challenging in practice because the boundaries of coronary artery calcium, the small lesions with large shape variation, are very blurry, resulting in poor performance in existing studies. To tackle this challenge, we present a novel Attention-based Multi-Scale Network called AMSN, which can process information through both the main and boundary branches in parallel. Key to our AMSN is a new non-local multi-scale context encoder module, which is mainly composed of the multi-scale attention mechanism and local global long short-term memory module. By aggregating the multi-scale context information, i.e. high-resolution low-level and low-resolution high-level features, the model's feature representative capability and deployment ability are improved effectively. Besides, we introduce a new boundary preserving loss, which can consider the boundary information of all coronary artery calcium together and establish links for the segmentation of different coronary artery calcium simultaneously. Extensive experiments demonstrate our AMSN enables reliable accurate coronary artery calcium segmentation for assisted cardiovascular disease diagnosis clinically.

Multi-Scale Attention Network for Diabetic Retinopathy Classification

Diabetic Retinopathy (DR) is a highly prevalent complication of diabetes mellitus, which causes lesions on the retina that affect vision which may lead to blindness if it is not detected and diagnosed early. Convolutional neural networks (CNN) are becoming the state-of-the-art approach for automatic detection of DR by using fundus images. The high-level features extracted by CNN are mostly utilised for the detection and classification of lesions on the retina. This high-level representation is capable of classifying different DR classes; however, more effective features for detecting the damages are needed. This paper proposes the multi-scale attention network (MSA-Net) for DR classification. The proposed approach applies the encoder network to embed the retina image in a high-level representational space, where the combination of mid and high-level features is used to enrich the representation. Then a multi-scale feature pyramid is included to describe the retinal structure in a different locality. Furthermore, to enhance the discriminative power of the feature representation a multi-scale attention mechanism is used on top of the high-level representation. The model is trained in a standard way using the cross-entropy loss to classify the DR severity level. In parallel as an auxiliary task, the model is trained using the weakly annotated data to detect healthy and non-healthy retina images. This surrogate task helps the model to enrich its discriminative power for distinguishing the non-healthy retina images. The proposed method when implemented has achieved outstanding results on two public datasets: EyePACS and APTOS.

U2-ONet: A Two-Level Nested Octave U-Structure Network with a Multi-Scale Attention Mechanism for Moving Object Segmentation

Most scenes in practical applications are dynamic scenes containing moving objects, so accurately segmenting moving objects is crucial for many computer vision applications. In order to efficiently segment all the moving objects in the scene, regardless of whether the object has a predefined semantic label, we propose a two-level nested octave U-structure network with a multi-scale attention mechanism, called U2-ONet. U2-ONet takes two RGB frames, the optical flow between these frames, and the instance segmentation of the frames as inputs. Each stage of U2-ONet is filled with the newly designed octave residual U-block (ORSU block) to enhance the ability to obtain more contextual information at different scales while reducing the spatial redundancy of the feature maps. In order to efficiently train the multi-scale deep network, we introduce a hierarchical training supervision strategy that calculates the loss at each level while adding knowledge-matching loss to keep the optimization consistent. The experimental results show that the proposed U2-ONet method can achieve a state-of-the-art performance in several general moving object segmentation datasets.

Regional registration of whole slide image stacks containing major histological artifacts

BackgroundHigh resolution 2D whole slide imaging provides rich information about the tissue structure. This information can be a lot richer if these 2D images can be stacked into a 3D tissue volume. A 3D analysis, however, requires accurate reconstruction of the tissue volume from the 2D image stack. This task is not trivial due to the distortions such as tissue tearing, folding and missing at each slide. Performing registration for the whole tissue slices may be adversely affected by distorted tissue regions. Consequently, regional registration is found to be more effective. In this paper, we propose a new approach to an accurate and robust registration of regions of interest for whole slide images. We introduce the idea of multi-scale attention for registration.ResultsUsing mean similarity index as the metric, the proposed algorithm (mean ± SD 0.84 pm 0.11) followed by a fine registration algorithm (0.86 pm 0.08) outperformed the state-of-the-art linear whole tissue registration algorithm (0.74 pm 0.19) and the regional version of this algorithm (0.81 pm 0.15). The proposed algorithm also outperforms the state-of-the-art nonlinear registration algorithm (original: 0.82 pm 0.12, regional: 0.77 pm 0.22) for whole slide images and a recently proposed patch-based registration algorithm (patch size 256: 0.79 pm 0.16 , patch size 512: 0.77 pm 0.16) for medical images.ConclusionUsing multi-scale attention mechanism leads to a more robust and accurate solution to the problem of regional registration of whole slide images corrupted in some parts by major histological artifacts in the imaged tissue.

Multiscale Omnibearing Attention Networks for Person Re-Identification

The past few years in the fields of Person Re-Identification (RE-ID) have seen attention mechanism receives enormous interest as it has superior performance in obtaining discriminative feature representations. However, a wide range of state-of-the-art RE-ID attention models only focus on one-dimensional attention design method, e.g. spatial attention and channels attention, hence the produced attention maps are neither detailed enough nor discriminative enough to capture complicated interactions of visual parts. Developing multi-scale attention mechanism for RE-ID, an under-studied approach, becomes a practicable method to overcome this deficiency. Toward this goal, we propose a Multiscale Omnibearing Attention Networks (MOAN) for RE-ID which is capable of utilizing the complex fusion information acquired from the multiscale attention mechanism with features being more representative. Specifically, MOAN takes full advantage of multi-sized convolution filters to obtain discriminative holistic and local feature maps, and adaptively conducts feature information augmentation by introducing an Omnibearing Attention (OA) module. Through the OA module, spatial attention and channel attention are integrated together in a unique way where they work in a complementary way. To sum up, MOAN not only inherits the merit of two kinds of attention mechanism but also performs well in extracting comprehensive feature information. Furthermore, taking into account the robustness of model performance, we formulate a Random Drop (RD) Function to facilitate training MOAN and further increase the diversity of training model for adaptation. Furthermore, to achieve end-to-end training, we utilize trainable parameters to take place of initial fixed parameters, and the model performance is experimentally promoted. Extensive experiments have been carried out on the four mainstream RE-ID datasets. As the result shows, our method with re-ranking achieves rank-1 accuracy of 92.29% on CUHK03-NP, 97.45% on Market-1501, 93.81% on DukeMTMC-reID and 81.53% on MSMT17-V2, outperforming the state-of-the-art methods and confirming the effectiveness of our method.

AttentionFGAN: Infrared and Visible Image Fusion Using Attention-Based Generative Adversarial Networks

Infrared and visible image fusion aims to describe the same scene from different aspects by combining complementary information of multi-modality images. The existing Generative adversarial networks (GAN) based infrared and visible image fusion methods cannot perceive the most discriminative regions, and hence fail to highlight the typical parts existing in infrared and visible images. To this end, we integrate multi-scale attention mechanism into both generator and discriminator of GAN to fuse infrared and visible images (AttentionFGAN). The multi-scale attention mechanism aims to not only capture comprehensive spatial information to help generator focus on the foreground target information of infrared image and background detail information of visible image, but also constrain the discriminators focus more on the attention regions rather than the whole input image. The generator of AttentionFGAN consists of two multi-scale attention networks and an image fusion network. Two multi-scale attention networks capture the attention maps of infrared and visible images respectively, so that the fusion network can reconstruct the fused image by paying more attention to the typical regions of source images. Besides, two discriminators are adopted to force the fused result keep more intensity and texture information from infrared and visible image respectively. Moreover, to keep more information of attention region from source images, an attention loss function is designed. Finally, the ablation experiments illustrate the effectiveness of the key parts of our method, and extensive qualitative and quantitative experiments on three public datasets demonstrate the advantages and effectiveness of AttentionFGAN compared with the other state-of-the-art methods.